Nov 04 00:54:46.563469 ip-10-0-137-48 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 04 00:54:46.563481 ip-10-0-137-48 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 04 00:54:46.563490 ip-10-0-137-48 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 04 00:54:46.563881 ip-10-0-137-48 systemd[1]: Failed to start Kubernetes Kubelet. Nov 04 00:54:57.436039 ip-10-0-137-48 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 04 00:54:57.436050 ip-10-0-137-48 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6f8aa927c3a941e1bdb76c61df5001d0 -- Nov 04 00:56:58.690405 ip-10-0-137-48 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:56:59.116691 ip-10-0-137-48 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:59.116691 ip-10-0-137-48 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:56:59.116691 ip-10-0-137-48 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:59.116691 ip-10-0-137-48 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:56:59.120717 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120633 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:59.120717 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120714 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:59.120717 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120718 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:59.120717 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120722 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:59.120717 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120726 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120730 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120734 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120739 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120743 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120747 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120751 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120756 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120762 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120774 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120779 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120783 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120788 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120792 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120796 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120800 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120804 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120808 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120812 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:59.121014 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120816 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120820 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120824 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120828 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120832 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120836 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120840 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120844 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120848 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120853 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120857 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120862 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120867 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120871 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120877 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120881 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120887 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120891 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120895 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120899 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:59.121810 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120904 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120909 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120914 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120919 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120923 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120927 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120931 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120939 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120946 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120951 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120955 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120959 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120964 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120968 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120972 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120977 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120981 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120986 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120990 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:59.122647 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120994 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.120998 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121002 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121006 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121011 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121015 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121019 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121024 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121028 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121032 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121037 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121042 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121047 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121052 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121056 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121060 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121064 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121069 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121073 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121077 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:59.123471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121082 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121087 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121091 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121095 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121099 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121103 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121108 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121112 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121132 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121138 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121142 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121146 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121150 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121155 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121159 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121164 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121168 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121174 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121179 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:59.124340 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121183 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121188 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121192 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121197 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.121202 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122570 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122582 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122587 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122592 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122596 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122600 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122604 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122609 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122613 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122617 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122622 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122626 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122631 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122635 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122640 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:59.124875 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122644 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122648 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122652 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122657 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122661 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122665 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122669 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122674 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122678 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122683 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122688 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122692 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122697 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122702 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122706 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122710 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122714 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122720 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122725 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:59.125439 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122729 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122733 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122737 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122742 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122746 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122750 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122754 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122759 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122763 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122767 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122771 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122776 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122780 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122784 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122791 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122796 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122802 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122809 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122814 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:59.126304 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122819 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122823 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122828 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122832 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122836 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122840 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122844 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122849 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122853 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122858 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122862 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122866 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122871 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122876 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122880 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122884 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122888 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122892 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122897 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:59.127147 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122901 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122905 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122912 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122916 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122920 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122924 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122928 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122933 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122940 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122945 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122951 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122958 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122962 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122971 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122975 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122980 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122984 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122988 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:59.127827 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122993 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.122997 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123001 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123006 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123010 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123014 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123019 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123024 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123028 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123033 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123037 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123041 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123045 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123050 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123055 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123882 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123891 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123895 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123900 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123904 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:59.128310 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123909 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123914 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123919 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123923 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123928 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123932 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123937 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123941 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123945 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123949 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123953 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123958 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123962 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123966 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123970 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123975 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123979 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123983 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123987 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:59.128782 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123993 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.123997 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124001 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124005 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124009 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124013 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124017 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124021 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124026 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124030 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124034 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124041 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124048 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124053 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124058 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124062 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124066 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124070 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124075 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:59.129272 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124079 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124083 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124087 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124092 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124096 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124100 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124104 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124108 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124112 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124133 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124137 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124142 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124146 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124152 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124157 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124161 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124165 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124169 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124173 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124177 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:59.129725 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124181 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124187 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124191 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124196 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124200 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124204 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124208 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124213 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124217 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124221 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124226 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124232 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124237 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124242 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124247 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124251 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124255 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124260 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124264 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:59.130243 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124269 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124273 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124277 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124281 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124286 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124290 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124294 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124298 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124304 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124308 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124312 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124317 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124321 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124325 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124329 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124334 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124338 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124343 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124347 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124351 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:59.130729 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124356 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124360 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124364 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.124368 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125302 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125315 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125327 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125334 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125341 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125346 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125357 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125364 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125370 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125375 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125381 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125387 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125392 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125396 2578 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125401 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125406 2578 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125411 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125417 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125423 2578 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:56:59.131210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125428 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125433 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125438 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125443 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125450 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125454 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125459 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125464 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125471 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125477 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125482 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125487 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125491 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125498 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125503 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125508 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125513 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125518 2578 flags.go:64] FLAG: --enable-server="true" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125522 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125532 2578 flags.go:64] FLAG: --event-burst="100" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125537 2578 flags.go:64] FLAG: --event-qps="50" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125541 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125546 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125551 2578 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:56:59.131743 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125558 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125563 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125568 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125572 2578 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125578 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125582 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125587 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125595 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125600 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125605 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125609 2578 flags.go:64] FLAG: --feature-gates="" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125624 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125629 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125635 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125640 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125644 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125652 2578 flags.go:64] FLAG: --help="false" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125657 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125663 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125668 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125672 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125678 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125684 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:56:59.132391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125688 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125693 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125698 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125703 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125708 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125713 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125718 2578 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125723 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125727 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125732 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125736 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125741 2578 flags.go:64] FLAG: --lock-file="" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125746 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125751 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125756 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125765 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125771 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125777 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125782 2578 flags.go:64] FLAG: --logging-format="text" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125787 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125792 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125797 2578 flags.go:64] FLAG: --manifest-url="" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125802 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125809 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125814 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:56:59.132953 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125822 2578 flags.go:64] FLAG: --max-pods="110" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125827 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125832 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125837 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125841 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125846 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125851 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125856 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125868 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125873 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125878 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125883 2578 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125887 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125892 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125897 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125902 2578 flags.go:64] FLAG: --port="10250" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125906 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125911 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0235cc2b8b49474a0" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125916 2578 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125921 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125926 2578 flags.go:64] FLAG: --register-node="true" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125932 2578 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125938 2578 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125944 2578 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125948 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125954 2578 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:56:59.133638 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125961 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125966 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125971 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125975 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125980 2578 flags.go:64] FLAG: --runonce="false" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125985 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125991 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.125996 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126001 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126006 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126011 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126016 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126021 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126026 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126030 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126035 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126040 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126044 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126049 2578 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126054 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126062 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126067 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126072 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126078 2578 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126083 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:56:59.134563 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126087 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126092 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126097 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126102 2578 flags.go:64] FLAG: --v="2" Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126110 2578 flags.go:64] FLAG: --version="false" Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126134 2578 flags.go:64] FLAG: --vmodule="" Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126142 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126148 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126363 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126370 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126374 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126379 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126384 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126390 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126396 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126400 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126404 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126408 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126413 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126417 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126421 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:59.135227 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126425 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126430 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126434 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126439 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126443 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126447 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126451 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126456 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126460 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126464 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126468 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126473 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126481 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126486 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126491 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126496 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126500 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126504 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:59.135703 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126509 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126515 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126519 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126524 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126529 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126535 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126540 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126544 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126548 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126552 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126557 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126561 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126566 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126570 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126575 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126581 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126588 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126593 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126598 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:59.136198 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126603 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126607 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126612 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126616 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126620 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126625 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126629 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126633 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126637 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126643 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126650 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126656 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126660 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126664 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126669 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126674 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126679 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126683 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126689 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:59.136734 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126694 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126698 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126703 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126707 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126711 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126716 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126720 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126724 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126728 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126733 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126737 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126741 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126746 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126750 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126754 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126758 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126762 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126767 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126771 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126775 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:59.137206 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126780 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126784 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126790 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126794 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126799 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126803 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126808 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126812 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126818 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126822 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126826 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126832 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126836 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126840 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126845 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126849 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:59.137687 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.126890 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126946 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126954 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126959 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126965 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126970 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126974 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126978 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126983 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126987 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126992 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.126996 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127000 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127005 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127009 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127013 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127018 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127022 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127026 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127032 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:59.138069 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127037 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127041 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127045 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127050 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127054 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127060 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127064 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127070 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127074 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127079 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127083 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127087 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127091 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127096 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127100 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127104 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127108 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127113 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127133 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:59.138554 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127138 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127143 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127147 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127151 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127156 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127160 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127164 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127168 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127172 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127176 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127181 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127185 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127194 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127201 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127207 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127212 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127216 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127220 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127225 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127231 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:59.139005 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127237 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127241 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127246 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127250 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127254 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127258 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127264 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127271 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127276 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127281 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127287 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127292 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127296 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127303 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127307 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127312 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127316 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127321 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127325 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:59.139546 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127329 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127333 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127338 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127342 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127346 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127353 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127358 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127362 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127366 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127371 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127375 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127380 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127385 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127391 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127395 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127400 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127404 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127408 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127413 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127417 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:59.139988 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127422 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127426 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127430 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127434 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127440 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127445 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127449 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127453 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.127457 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.134662 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.134679 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.134693 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.134698 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134764 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134769 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134772 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134775 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134778 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134781 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134784 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:59.140479 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134786 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134789 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134791 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134794 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134797 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134799 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134802 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134804 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134807 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134809 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134812 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134815 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134817 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134821 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134823 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134826 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134828 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134831 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134834 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134836 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:59.140946 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134839 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134841 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134846 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134851 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134854 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134857 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134859 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134862 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134864 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134867 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134870 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134872 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134875 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134877 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134881 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134883 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134886 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134888 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134891 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:59.141471 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134895 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134898 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134900 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134903 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134905 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134908 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134911 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134913 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134916 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134918 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134921 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134924 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134926 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134928 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134932 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134935 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134938 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134941 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134944 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134946 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:59.141921 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134949 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134953 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134957 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134961 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134964 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134966 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134969 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134972 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134975 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134977 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134980 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134983 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134985 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134988 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134991 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134994 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134996 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.134999 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135002 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:59.142414 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135004 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135007 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135009 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135012 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135014 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135017 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135020 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135022 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135026 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135028 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135031 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135034 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135037 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135039 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135042 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135044 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135047 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135049 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135052 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135055 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:59.142865 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135057 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.135084 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135248 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135254 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135257 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135261 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135264 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135267 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135270 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135273 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135276 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135279 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135282 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135285 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135287 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135290 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:59.143351 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135293 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135296 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135298 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135301 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135303 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135306 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135309 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135311 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135314 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135317 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135319 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135321 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135324 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135327 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135329 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135331 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135334 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135336 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135338 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135341 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:59.143724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135343 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135346 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135348 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135351 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135354 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135357 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135359 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135362 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135365 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135367 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135370 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135372 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135375 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135377 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135380 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135382 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135385 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135387 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135390 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135392 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:59.144191 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135395 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135397 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135400 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135403 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135405 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135408 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135410 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135413 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135415 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135417 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135420 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135422 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135425 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135428 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135430 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135433 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135436 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135439 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135441 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:59.144686 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135444 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135448 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135452 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135455 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135457 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135460 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135463 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135466 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135469 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135471 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135474 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135477 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135479 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135482 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135484 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135487 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135491 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135495 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135498 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:59.145149 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135500 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135503 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135505 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135508 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135510 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135513 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135515 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135518 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135520 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135523 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135525 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135528 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135531 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135534 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.135557 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135589 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:56:59.145631 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135593 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135596 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135599 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135602 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135604 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135606 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135609 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135611 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135614 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135616 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135619 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135621 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135624 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135626 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135629 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135631 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135634 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135636 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135639 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135641 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:56:59.146016 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135644 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135646 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135649 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135652 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135654 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135656 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135662 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135665 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135667 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135671 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135673 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135676 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135678 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135681 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135684 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135686 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135689 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135691 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135694 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:56:59.146511 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135696 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135699 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135701 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135704 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135706 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135709 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135712 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135714 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135717 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135719 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135722 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135725 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135728 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135730 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135732 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135735 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135737 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135740 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135742 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135746 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:56:59.146956 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135749 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135752 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135754 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135757 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135760 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135763 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135765 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135768 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135772 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135775 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135778 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135781 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135783 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135785 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135788 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135791 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135793 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135796 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135798 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:56:59.147428 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135801 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135803 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135806 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135808 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135812 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135816 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135819 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135822 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135827 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135829 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135833 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135835 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135839 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135842 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135845 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135847 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135850 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135853 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135856 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:56:59.147903 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135858 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135861 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135863 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135866 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135868 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135871 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135873 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:56:59.135876 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.135934 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.139863 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.140913 2578 server.go:1014] "Starting client certificate rotation" Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.141006 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.141040 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.142944 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:56:59.148423 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.147086 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:56:59.158803 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.158787 2578 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:56:59.162466 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.162452 2578 log.go:25] "Validated CRI v1 image API" Nov 04 00:56:59.163250 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.163229 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:56:59.166797 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.166774 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 940ded07-229b-49ff-8704-ce7ab1eec3f8:/dev/nvme0n1p4 9afb4279-08e1-43bd-a574-76d8bf9f4e8d:/dev/nvme0n1p3] Nov 04 00:56:59.166882 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.166794 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 00:56:59.169103 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.169088 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:56:59.171334 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.171226 2578 manager.go:217] Machine: {Timestamp:2026-11-04 00:56:59.170757057 +0000 UTC m=+0.359924485 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102303 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b83d2d941938a9da675ee145c52cd SystemUUID:ec2b83d2-d941-938a-9da6-75ee145c52cd BootID:6f8aa927-c3a9-41e1-bdb7-6c61df5001d0 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:a8:74:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:a8:74:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:88:c8:e0:e8:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:56:59.171334 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.171332 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:56:59.171438 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.171427 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:56:59.174958 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.174937 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:56:59.175080 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.174959 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-48.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:56:59.175133 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.175081 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:56:59.175133 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.175089 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:56:59.175133 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.175112 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:56:59.175213 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.175147 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:56:59.175827 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.175816 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:56:59.179317 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.179305 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:56:59.179431 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.179422 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:56:59.179474 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.179455 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:56:59.179474 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.179464 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:56:59.179474 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.179474 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:56:59.179606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.179483 2578 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:56:59.179606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.179491 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:56:59.179797 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.179783 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:56:59.183565 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.183543 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:56:59.187893 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.187880 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:56:59.188607 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.188595 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:56:59.188643 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.188625 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:56:59.188779 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.188768 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:56:59.188813 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.188784 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:56:59.188813 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.188791 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:56:59.188813 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.188796 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:56:59.188813 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.188802 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:56:59.188813 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.188807 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:56:59.189401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.189392 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:56:59.189446 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.189402 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:56:59.189446 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.189410 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:56:59.189446 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.189416 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:56:59.189446 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.189425 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:56:59.189446 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.189432 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:56:59.189706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.189450 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:56:59.189706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.189456 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:56:59.192293 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.192261 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-48.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 00:56:59.193235 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.193220 2578 server.go:1263] "Started kubelet" Nov 04 00:56:59.193310 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.193280 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:56:59.193366 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.193306 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:56:59.193406 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.193366 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:56:59.193896 ip-10-0-137-48 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:56:59.194611 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.194591 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:56:59.195707 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.195325 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:56:59.199053 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.199035 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 00:56:59.200669 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.200655 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:56:59.200732 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.200670 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:56:59.200797 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.200726 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:56:59.201455 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201310 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:56:59.201455 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201311 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:56:59.201455 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201398 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:56:59.201632 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201469 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:56:59.201632 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201479 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:56:59.201632 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201480 2578 factory.go:55] Registering systemd factory Nov 04 00:56:59.201632 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201495 2578 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:56:59.201749 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201731 2578 factory.go:153] Registering CRI-O factory Nov 04 00:56:59.201749 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201744 2578 factory.go:223] Registration of the crio container factory successfully Nov 04 00:56:59.201824 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201816 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:56:59.201857 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201838 2578 factory.go:103] Registering Raw factory Nov 04 00:56:59.201857 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.201852 2578 manager.go:1199] Started watching for new ooms in manager Nov 04 00:56:59.203281 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.203259 2578 manager.go:319] Starting recovery of all containers Nov 04 00:56:59.208676 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.208648 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wcl5k" Nov 04 00:56:59.208999 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.208966 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 00:56:59.209198 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.209176 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-48.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 00:56:59.209198 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.209175 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-48.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 00:56:59.209870 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.208838 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-48.us-east-2.compute.internal.18e4b1984782afc9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-48.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-48.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:56:59.193192393 +0000 UTC m=+0.382359823,LastTimestamp:2026-11-04 00:56:59.193192393 +0000 UTC m=+0.382359823,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-48.us-east-2.compute.internal,}" Nov 04 00:56:59.212216 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.212158 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 00:56:59.215348 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.215329 2578 manager.go:324] Recovery completed Nov 04 00:56:59.216414 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.216396 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wcl5k" Nov 04 00:56:59.221344 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.221328 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:59.226270 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.226250 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:59.226352 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.226281 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:59.226352 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.226292 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:59.227583 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.227570 2578 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:56:59.227583 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.227580 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:56:59.227696 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.227586 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:59.227696 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.227595 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:56:59.228649 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.228583 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-48.us-east-2.compute.internal.18e4b198497b60e8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-48.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-48.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-48.us-east-2.compute.internal,},FirstTimestamp:2026-11-04 00:56:59.22626788 +0000 UTC m=+0.415435308,LastTimestamp:2026-11-04 00:56:59.22626788 +0000 UTC m=+0.415435308,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-48.us-east-2.compute.internal,}" Nov 04 00:56:59.228836 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.228820 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:59.228899 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.228846 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:59.228899 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.228860 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:59.229761 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.229746 2578 policy_none.go:50] "Start" Nov 04 00:56:59.229809 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.229767 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:56:59.229809 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.229781 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:56:59.231889 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.231877 2578 policy_none.go:44] "Start" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.257738 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.260237 2578 manager.go:341] "Starting Device Plugin manager" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.260306 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.260318 2578 server.go:87] "Starting device plugin registration server" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.260545 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.260556 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.260610 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.260691 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.260698 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.261107 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.261202 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.268217 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.269305 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.269322 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.269341 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.269371 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 04 00:56:59.280706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.272387 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:56:59.361415 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.361386 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:59.362169 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.362153 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:59.362259 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.362181 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:59.362259 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.362193 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:59.362259 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.362219 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.370511 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.370461 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal"] Nov 04 00:56:59.370585 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.370513 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:59.371237 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.371222 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:59.371300 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.371248 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:59.371300 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.371258 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:59.371712 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.371699 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.371752 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.371718 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-48.us-east-2.compute.internal\": node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:56:59.376042 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376030 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:59.376182 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376170 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.376223 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376199 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:59.376787 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376763 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:59.376867 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376805 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:59.376867 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376818 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:59.376867 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376827 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:59.376867 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376843 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:59.376867 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.376843 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:59.379046 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.379030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.379111 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.379056 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:56:59.379795 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.379777 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:56:59.379871 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.379804 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:56:59.379871 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.379818 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:56:59.398698 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.398679 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:56:59.402726 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.402674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60a4a4e0e49879528cdf55442dd62356-config\") pod \"kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"60a4a4e0e49879528cdf55442dd62356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.402726 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.402698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.402726 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.402715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.405610 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.405594 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" node="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.407612 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.407598 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" node="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.499374 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.499347 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:56:59.503671 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.503652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60a4a4e0e49879528cdf55442dd62356-config\") pod \"kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"60a4a4e0e49879528cdf55442dd62356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.503746 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.503677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.503746 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.503694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.503746 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.503719 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.503838 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.503742 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60a4a4e0e49879528cdf55442dd62356-config\") pod \"kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"60a4a4e0e49879528cdf55442dd62356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.503838 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.503749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.599845 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.599810 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:56:59.700578 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.700528 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:56:59.708887 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.708870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.711026 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:56:59.711013 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:56:59.801180 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.801024 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:56:59.901479 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:56:59.901453 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:57:00.001852 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:00.001798 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:57:00.060061 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.060042 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:00.087531 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.087515 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:00.102640 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:00.102622 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:57:00.140847 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.140830 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 00:57:00.141222 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.140911 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:57:00.141222 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.140950 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:57:00.141222 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.140969 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 04 00:57:00.203455 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:00.203439 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:57:00.221557 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.221530 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:51:58 +0000 UTC" deadline="2028-05-16 16:52:32.600420264 +0000 UTC" Nov 04 00:57:00.221557 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.221553 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13431h55m32.378870119s" Nov 04 00:57:00.304283 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:00.304259 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:57:00.379928 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:00.379893 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a4a4e0e49879528cdf55442dd62356.slice/crio-05fc03a562394dad9dcc7c076bd724c757e3abc030e4aacc92b620e9fcb4de6e WatchSource:0}: Error finding container 05fc03a562394dad9dcc7c076bd724c757e3abc030e4aacc92b620e9fcb4de6e: Status 404 returned error can't find the container with id 05fc03a562394dad9dcc7c076bd724c757e3abc030e4aacc92b620e9fcb4de6e Nov 04 00:57:00.380151 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:00.380132 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a84db2d45c5cb12625222b3483c125f.slice/crio-8d1a583db533a1b477e566ec28c586797b3898f1ec1e2c2d68af2bfbb1c58942 WatchSource:0}: Error finding container 8d1a583db533a1b477e566ec28c586797b3898f1ec1e2c2d68af2bfbb1c58942: Status 404 returned error can't find the container with id 8d1a583db533a1b477e566ec28c586797b3898f1ec1e2c2d68af2bfbb1c58942 Nov 04 00:57:00.385665 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.385652 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:57:00.404563 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:00.404546 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-48.us-east-2.compute.internal\" not found" Nov 04 00:57:00.457101 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.457082 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:00.500968 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.500947 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:00.513209 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.513165 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:00.514185 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.514174 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:00.521898 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:00.521887 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:01.180737 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.180708 2578 apiserver.go:52] "Watching apiserver" Nov 04 00:57:01.190580 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.189955 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:57:01.190580 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.190301 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-8rwkl","openshift-image-registry/node-ca-dcksb","openshift-multus/multus-dm58x","openshift-network-diagnostics/network-check-target-zh5bh","openshift-ovn-kubernetes/ovnkube-node-j49c7","openshift-cluster-node-tuning-operator/tuned-c4pg2","openshift-dns/node-resolver-shjsf","openshift-multus/multus-additional-cni-plugins-rlwsw","openshift-multus/network-metrics-daemon-m84bl","openshift-network-operator/iptables-alerter-fkvdp","kube-system/konnectivity-agent-w6tt8","kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal"] Nov 04 00:57:01.192595 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.192574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.195007 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.194535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.196138 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.195939 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.196138 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.195988 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:57:01.196138 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.196020 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:57:01.196138 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.196094 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.196706 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.196690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.197679 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.197661 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.197843 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.197824 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.197920 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.197884 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:57:01.198688 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.198670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.199448 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.199431 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:57:01.200087 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.200061 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:57:01.200906 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.200893 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.201319 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.201297 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.201557 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.201541 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:57:01.201891 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.201877 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.201945 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.201916 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 00:57:01.202955 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.202937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:01.203032 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.203011 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:01.203695 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.203667 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.203695 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.203675 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.204844 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.204829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.207212 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.207196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.209875 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.209858 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.211153 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211134 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:57:01.211250 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211161 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:57:01.211250 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211172 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sknxk\"" Nov 04 00:57:01.211250 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211194 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgrbh\"" Nov 04 00:57:01.211380 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211290 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqwbk\"" Nov 04 00:57:01.211380 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211288 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xdt54\"" Nov 04 00:57:01.211468 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211378 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ln4qv\"" Nov 04 00:57:01.211515 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211493 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9jwd5\"" Nov 04 00:57:01.211560 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.211511 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zx855\"" Nov 04 00:57:01.212143 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212110 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.212233 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.212178 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:01.212322 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.212401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-system-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212482 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212482 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-cnibin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212482 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-conf-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212482 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbb6\" (UniqueName: \"kubernetes.io/projected/0c8ada62-5c11-4462-a29d-a576d7c162df-kube-api-access-mgbb6\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.212692 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212517 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-host\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.212692 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-netns\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212692 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212561 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4k8lm\"" Nov 04 00:57:01.212692 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-kubelet\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212692 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-hostroot\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212692 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-socket-dir-parent\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212692 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-k8s-cni-cncf-io\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3391f8ba-2fe6-4950-b644-5361fc8c2e48-host-slash\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212765 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-binary-copy\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-serviceca\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-bin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-multus-certs\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212892 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-registration-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.212991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.212985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-sys-fs\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95cq\" (UniqueName: \"kubernetes.io/projected/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kube-api-access-m95cq\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-etc-kubernetes\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3391f8ba-2fe6-4950-b644-5361fc8c2e48-iptables-alerter-script\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhbb\" (UniqueName: \"kubernetes.io/projected/3391f8ba-2fe6-4950-b644-5361fc8c2e48-kube-api-access-kfhbb\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-device-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-system-cni-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxj6\" (UniqueName: \"kubernetes.io/projected/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-kube-api-access-htxj6\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-cnibin\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-socket-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-cni-binary-copy\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-multus\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.213401 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-multus-daemon-config\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.214051 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213401 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl6xd\" (UniqueName: \"kubernetes.io/projected/08221c6a-1a73-4922-a769-caeb42c194bb-kube-api-access-gl6xd\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.214051 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.213433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.214904 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.214885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:01.214985 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.214931 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:01.217100 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.217079 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:01.229332 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.229293 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rkzgq" Nov 04 00:57:01.236077 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.236061 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rkzgq" Nov 04 00:57:01.274285 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.274244 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"60a4a4e0e49879528cdf55442dd62356","Type":"ContainerStarted","Data":"05fc03a562394dad9dcc7c076bd724c757e3abc030e4aacc92b620e9fcb4de6e"} Nov 04 00:57:01.275304 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.275283 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"1a84db2d45c5cb12625222b3483c125f","Type":"ContainerStarted","Data":"8d1a583db533a1b477e566ec28c586797b3898f1ec1e2c2d68af2bfbb1c58942"} Nov 04 00:57:01.288214 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288188 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:57:01.288312 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288228 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.288312 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288265 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.288312 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288293 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.288458 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288329 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:57:01.288458 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288347 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:57:01.288458 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288389 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.288458 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288398 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:57:01.288458 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288424 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:57:01.288647 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.288511 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:57:01.297185 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.297167 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:01.302018 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.302002 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-psr9r\"" Nov 04 00:57:01.302093 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.302077 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:57:01.302471 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.302454 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:57:01.313861 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-multus-certs\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.313861 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-registration-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.313861 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-multus-certs\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.313861 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95cq\" (UniqueName: \"kubernetes.io/projected/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kube-api-access-m95cq\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.313861 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313819 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-env-overrides\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.313861 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75b2w\" (UniqueName: \"kubernetes.io/projected/1006149a-92ee-43c1-bb41-6aaf1df41c36-kube-api-access-75b2w\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.313861 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313821 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-registration-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-etc-kubernetes\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313935 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-etc-kubernetes\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.313960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-modprobe-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-tmp\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-agent-certs\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-system-cni-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxj6\" (UniqueName: \"kubernetes.io/projected/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-kube-api-access-htxj6\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-run\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-host\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.314179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-system-cni-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-systemd\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314282 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlml8\" (UniqueName: \"kubernetes.io/projected/5b2c9231-5813-4143-93ca-2457278f37a2-kube-api-access-nlml8\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-cnibin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysconfig\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-kubelet-config\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-netns\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-node-log\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-var-lib-kubelet\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-kubelet\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.314606 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-hostroot\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-k8s-cni-cncf-io\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-cnibin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-hostroot\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3391f8ba-2fe6-4950-b644-5361fc8c2e48-host-slash\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-binary-copy\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-serviceca\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-bin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-sys-fs\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-kubelet\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-systemd-units\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314894 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-bin\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314909 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-netd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3391f8ba-2fe6-4950-b644-5361fc8c2e48-iptables-alerter-script\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.315363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhbb\" (UniqueName: \"kubernetes.io/projected/3391f8ba-2fe6-4950-b644-5361fc8c2e48-kube-api-access-kfhbb\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-device-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-tuned\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.314984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51ff9909-e56c-4d27-be31-574bf33a8f75-hosts-file\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ff9909-e56c-4d27-be31-574bf33a8f75-tmp-dir\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-netns\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-sys-fs\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3391f8ba-2fe6-4950-b644-5361fc8c2e48-host-slash\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-k8s-cni-cncf-io\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315255 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-kubelet\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315286 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-netns\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-script-lib\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjjp\" (UniqueName: \"kubernetes.io/projected/51ff9909-e56c-4d27-be31-574bf33a8f75-kube-api-access-9cjjp\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-konnectivity-ca\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-cnibin\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.316020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-socket-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-lib-modules\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-dbus\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-config\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-sys\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-cni-binary-copy\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315663 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-binary-copy\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-multus\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-multus-daemon-config\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315745 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-bin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl6xd\" (UniqueName: \"kubernetes.io/projected/08221c6a-1a73-4922-a769-caeb42c194bb-kube-api-access-gl6xd\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-kubernetes\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-conf\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-system-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-conf-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.316719 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbb6\" (UniqueName: \"kubernetes.io/projected/0c8ada62-5c11-4462-a29d-a576d7c162df-kube-api-access-mgbb6\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-host\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-etc-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-log-socket\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovn-node-metrics-cert\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sb2n\" (UniqueName: \"kubernetes.io/projected/51fbfff7-c90b-4726-bbff-6ebbcc943626-kube-api-access-6sb2n\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-systemd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-var-lib-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-socket-dir-parent\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-slash\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316099 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-ovn\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.315997 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-serviceca\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-cni-binary-copy\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316577 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-multus-daemon-config\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-socket-dir-parent\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.317462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316692 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316821 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-system-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-conf-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.316944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-socket-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.317016 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-device-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.317054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-cnibin\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.317078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-multus\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.317085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.317088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3391f8ba-2fe6-4950-b644-5361fc8c2e48-iptables-alerter-script\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.317969 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.317157 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-host\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.320939 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.320915 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:57:01.322004 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.321982 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:01.322087 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.322008 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:01.322087 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.322026 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xswcp for pod openshift-network-diagnostics/network-check-target-zh5bh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:01.322211 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.322102 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp podName:66c16477-f3db-43cf-9830-39fe0267bdce nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.822083667 +0000 UTC m=+3.011251099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xswcp" (UniqueName: "kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp") pod "network-check-target-zh5bh" (UID: "66c16477-f3db-43cf-9830-39fe0267bdce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:01.326137 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.325535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxj6\" (UniqueName: \"kubernetes.io/projected/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-kube-api-access-htxj6\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.327517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.327485 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhbb\" (UniqueName: \"kubernetes.io/projected/3391f8ba-2fe6-4950-b644-5361fc8c2e48-kube-api-access-kfhbb\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.327610 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.327570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbb6\" (UniqueName: \"kubernetes.io/projected/0c8ada62-5c11-4462-a29d-a576d7c162df-kube-api-access-mgbb6\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.327831 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.327810 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95cq\" (UniqueName: \"kubernetes.io/projected/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kube-api-access-m95cq\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.328399 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.328382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl6xd\" (UniqueName: \"kubernetes.io/projected/08221c6a-1a73-4922-a769-caeb42c194bb-kube-api-access-gl6xd\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.417311 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417311 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-systemd\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417396 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-systemd\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.417447 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlml8\" (UniqueName: \"kubernetes.io/projected/5b2c9231-5813-4143-93ca-2457278f37a2-kube-api-access-nlml8\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysconfig\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.417505 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.917487137 +0000 UTC m=+3.106654565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.417502 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:01.417521 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-kubelet-config\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-node-log\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.417576 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret podName:04f82b22-3949-44d3-abe9-6473a2f68f76 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:01.917558314 +0000 UTC m=+3.106725741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret") pod "global-pull-secret-syncer-8rwkl" (UID: "04f82b22-3949-44d3-abe9-6473a2f68f76") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-node-log\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysconfig\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-var-lib-kubelet\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417632 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-kubelet-config\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417656 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-var-lib-kubelet\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-kubelet\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-systemd-units\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417692 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-kubelet\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-bin\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-netd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417732 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-systemd-units\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-tuned\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417768 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-bin\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.417993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417771 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-netd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51ff9909-e56c-4d27-be31-574bf33a8f75-hosts-file\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ff9909-e56c-4d27-be31-574bf33a8f75-tmp-dir\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51ff9909-e56c-4d27-be31-574bf33a8f75-hosts-file\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-netns\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-script-lib\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417866 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-netns\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjjp\" (UniqueName: \"kubernetes.io/projected/51ff9909-e56c-4d27-be31-574bf33a8f75-kube-api-access-9cjjp\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-konnectivity-ca\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-lib-modules\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-dbus\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.417982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-config\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-sys\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-kubernetes\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-conf\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-dbus\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-etc-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-etc-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.418552 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-log-socket\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-kubernetes\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-log-socket\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-sys\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418250 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-conf\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovn-node-metrics-cert\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418280 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ff9909-e56c-4d27-be31-574bf33a8f75-tmp-dir\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sb2n\" (UniqueName: \"kubernetes.io/projected/51fbfff7-c90b-4726-bbff-6ebbcc943626-kube-api-access-6sb2n\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-lib-modules\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-systemd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-var-lib-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-slash\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-ovn\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-env-overrides\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b2w\" (UniqueName: \"kubernetes.io/projected/1006149a-92ee-43c1-bb41-6aaf1df41c36-kube-api-access-75b2w\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-modprobe-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-tmp\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.419335 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-agent-certs\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418560 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-script-lib\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-run\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418576 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-host\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-run\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418646 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-konnectivity-ca\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418695 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-modprobe-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-systemd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-host\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-ovn\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.418982 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-var-lib-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.419015 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.419018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-slash\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.419172 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-config\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.420744 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.420416 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-env-overrides\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.421662 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.421445 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-tmp\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.421662 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.421462 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-tuned\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.421662 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.421587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-agent-certs\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:01.424745 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.424511 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovn-node-metrics-cert\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.427277 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.427254 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b2w\" (UniqueName: \"kubernetes.io/projected/1006149a-92ee-43c1-bb41-6aaf1df41c36-kube-api-access-75b2w\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.428651 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.428634 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjjp\" (UniqueName: \"kubernetes.io/projected/51ff9909-e56c-4d27-be31-574bf33a8f75-kube-api-access-9cjjp\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.428788 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.428768 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlml8\" (UniqueName: \"kubernetes.io/projected/5b2c9231-5813-4143-93ca-2457278f37a2-kube-api-access-nlml8\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:01.430003 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.429984 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sb2n\" (UniqueName: \"kubernetes.io/projected/51fbfff7-c90b-4726-bbff-6ebbcc943626-kube-api-access-6sb2n\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.510141 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.510055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dm58x" Nov 04 00:57:01.515818 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.515787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:01.518805 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:01.518771 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08221c6a_1a73_4922_a769_caeb42c194bb.slice/crio-f5b036939fa5a162d05daba808cab6c16ba1a7d3bc1debd8ed08861578b20104 WatchSource:0}: Error finding container f5b036939fa5a162d05daba808cab6c16ba1a7d3bc1debd8ed08861578b20104: Status 404 returned error can't find the container with id f5b036939fa5a162d05daba808cab6c16ba1a7d3bc1debd8ed08861578b20104 Nov 04 00:57:01.524060 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:01.524036 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3391f8ba_2fe6_4950_b644_5361fc8c2e48.slice/crio-570379eee1ef601a71e6a50ba4e92747a095999d05d12476a1256eaf47e86abd WatchSource:0}: Error finding container 570379eee1ef601a71e6a50ba4e92747a095999d05d12476a1256eaf47e86abd: Status 404 returned error can't find the container with id 570379eee1ef601a71e6a50ba4e92747a095999d05d12476a1256eaf47e86abd Nov 04 00:57:01.536039 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.536020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:01.539175 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.539148 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:01.543081 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.542936 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:01.549841 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:01.549769 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8ada62_5c11_4462_a29d_a576d7c162df.slice/crio-06235872e24e5f1763a8545a9e07d1730cab3c58732793911f5888e5a007bde9 WatchSource:0}: Error finding container 06235872e24e5f1763a8545a9e07d1730cab3c58732793911f5888e5a007bde9: Status 404 returned error can't find the container with id 06235872e24e5f1763a8545a9e07d1730cab3c58732793911f5888e5a007bde9 Nov 04 00:57:01.553011 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:01.552986 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a1b8a9_e616_4cb0_9e1e_ce3ff75251f7.slice/crio-93a338d9cb30e8d5fe70d82c4d8485b1df68152bf391f6d029caa34d4ae5d383 WatchSource:0}: Error finding container 93a338d9cb30e8d5fe70d82c4d8485b1df68152bf391f6d029caa34d4ae5d383: Status 404 returned error can't find the container with id 93a338d9cb30e8d5fe70d82c4d8485b1df68152bf391f6d029caa34d4ae5d383 Nov 04 00:57:01.556598 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.556581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:01.562363 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.562342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:01.564757 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.564738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:01.566306 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:01.566285 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fbfff7_c90b_4726_bbff_6ebbcc943626.slice/crio-97b3a3120c6873e725bc59def220448c6613aa0da419b12eaf3b694167b37d86 WatchSource:0}: Error finding container 97b3a3120c6873e725bc59def220448c6613aa0da419b12eaf3b694167b37d86: Status 404 returned error can't find the container with id 97b3a3120c6873e725bc59def220448c6613aa0da419b12eaf3b694167b37d86 Nov 04 00:57:01.566393 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.566341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:01.572415 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:01.572389 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1006149a_92ee_43c1_bb41_6aaf1df41c36.slice/crio-0bbaee0ae0cd851dba5ede4a2f5c3c7cffc339fd7604199922db1c988f232812 WatchSource:0}: Error finding container 0bbaee0ae0cd851dba5ede4a2f5c3c7cffc339fd7604199922db1c988f232812: Status 404 returned error can't find the container with id 0bbaee0ae0cd851dba5ede4a2f5c3c7cffc339fd7604199922db1c988f232812 Nov 04 00:57:01.576204 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:01.576164 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ff9909_e56c_4d27_be31_574bf33a8f75.slice/crio-ab74b5ef9f2a362a896fc842c43c761ced299b84f83cd0d663eb9e1527c433b0 WatchSource:0}: Error finding container ab74b5ef9f2a362a896fc842c43c761ced299b84f83cd0d663eb9e1527c433b0: Status 404 returned error can't find the container with id ab74b5ef9f2a362a896fc842c43c761ced299b84f83cd0d663eb9e1527c433b0 Nov 04 00:57:01.578834 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:01.578818 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef08de9_97a2_4c5b_ae3a_66a9cd783a31.slice/crio-cc42596f360aaadedf53aa8805a79bd93fdf2374aa451a6fe4460495ea0ef2e1 WatchSource:0}: Error finding container cc42596f360aaadedf53aa8805a79bd93fdf2374aa451a6fe4460495ea0ef2e1: Status 404 returned error can't find the container with id cc42596f360aaadedf53aa8805a79bd93fdf2374aa451a6fe4460495ea0ef2e1 Nov 04 00:57:01.903612 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.903587 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:01.921972 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.921946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:01.922129 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.922096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:01.922129 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.922108 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:01.922214 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:01.922139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:01.922214 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.922144 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:01.922214 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.922154 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xswcp for pod openshift-network-diagnostics/network-check-target-zh5bh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:01.922214 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.922194 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp podName:66c16477-f3db-43cf-9830-39fe0267bdce nodeName:}" failed. No retries permitted until 2026-11-04 00:57:02.922181613 +0000 UTC m=+4.111349030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xswcp" (UniqueName: "kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp") pod "network-check-target-zh5bh" (UID: "66c16477-f3db-43cf-9830-39fe0267bdce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:01.922343 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.922227 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:01.922343 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.922260 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:02.922250508 +0000 UTC m=+4.111417950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:01.922343 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.922292 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:01.922343 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:01.922310 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret podName:04f82b22-3949-44d3-abe9-6473a2f68f76 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:02.922304364 +0000 UTC m=+4.111471798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret") pod "global-pull-secret-syncer-8rwkl" (UID: "04f82b22-3949-44d3-abe9-6473a2f68f76") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:02.239652 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.239257 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:52:00 +0000 UTC" deadline="2028-05-05 04:25:54.611059422 +0000 UTC" Nov 04 00:57:02.239652 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.239549 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13155h28m52.371516884s" Nov 04 00:57:02.270862 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.270527 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:02.270862 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.270612 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:02.288322 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.288227 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fkvdp" event={"ID":"3391f8ba-2fe6-4950-b644-5361fc8c2e48","Type":"ContainerStarted","Data":"570379eee1ef601a71e6a50ba4e92747a095999d05d12476a1256eaf47e86abd"} Nov 04 00:57:02.290113 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.290077 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dm58x" event={"ID":"08221c6a-1a73-4922-a769-caeb42c194bb","Type":"ContainerStarted","Data":"f5b036939fa5a162d05daba808cab6c16ba1a7d3bc1debd8ed08861578b20104"} Nov 04 00:57:02.292347 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.292316 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-shjsf" event={"ID":"51ff9909-e56c-4d27-be31-574bf33a8f75","Type":"ContainerStarted","Data":"ab74b5ef9f2a362a896fc842c43c761ced299b84f83cd0d663eb9e1527c433b0"} Nov 04 00:57:02.295828 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.295807 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"0bbaee0ae0cd851dba5ede4a2f5c3c7cffc339fd7604199922db1c988f232812"} Nov 04 00:57:02.298080 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.297945 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" event={"ID":"51fbfff7-c90b-4726-bbff-6ebbcc943626","Type":"ContainerStarted","Data":"97b3a3120c6873e725bc59def220448c6613aa0da419b12eaf3b694167b37d86"} Nov 04 00:57:02.306261 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.306235 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" event={"ID":"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7","Type":"ContainerStarted","Data":"93a338d9cb30e8d5fe70d82c4d8485b1df68152bf391f6d029caa34d4ae5d383"} Nov 04 00:57:02.315026 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.315002 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"60a4a4e0e49879528cdf55442dd62356","Type":"ContainerStarted","Data":"af1966b41412f5645ac46a26f5458e3dca82c3baaf8cc8887e1f02f97e1f9cd2"} Nov 04 00:57:02.334737 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.333983 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a84db2d45c5cb12625222b3483c125f" containerID="b73c9d32e77a52cd15af814a31b9c0212466de5ef3bef800276c9b597fcecc29" exitCode=0 Nov 04 00:57:02.334737 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.334062 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"1a84db2d45c5cb12625222b3483c125f","Type":"ContainerDied","Data":"b73c9d32e77a52cd15af814a31b9c0212466de5ef3bef800276c9b597fcecc29"} Nov 04 00:57:02.344947 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.344865 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w6tt8" event={"ID":"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31","Type":"ContainerStarted","Data":"cc42596f360aaadedf53aa8805a79bd93fdf2374aa451a6fe4460495ea0ef2e1"} Nov 04 00:57:02.349716 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.349044 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" podStartSLOduration=2.349028601 podStartE2EDuration="2.349028601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:02.329545174 +0000 UTC m=+3.518712609" watchObservedRunningTime="2026-11-04 00:57:02.349028601 +0000 UTC m=+3.538196038" Nov 04 00:57:02.350309 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.350280 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" event={"ID":"0c8ada62-5c11-4462-a29d-a576d7c162df","Type":"ContainerStarted","Data":"06235872e24e5f1763a8545a9e07d1730cab3c58732793911f5888e5a007bde9"} Nov 04 00:57:02.362391 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.362342 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcksb" event={"ID":"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56","Type":"ContainerStarted","Data":"7dafca7abc9eb105558fa198a460f6200041eeaed8974ca72bf3928f5a65d0fb"} Nov 04 00:57:02.426645 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.426540 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:02.929188 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.929098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:02.929188 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.929163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:02.929399 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:02.929199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:02.929399 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.929325 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:02.929399 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.929384 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:02.929399 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.929392 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret podName:04f82b22-3949-44d3-abe9-6473a2f68f76 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:04.929375026 +0000 UTC m=+6.118542444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret") pod "global-pull-secret-syncer-8rwkl" (UID: "04f82b22-3949-44d3-abe9-6473a2f68f76") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:02.929399 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.929401 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:02.929622 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.929413 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xswcp for pod openshift-network-diagnostics/network-check-target-zh5bh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:02.929622 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.929457 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp podName:66c16477-f3db-43cf-9830-39fe0267bdce nodeName:}" failed. No retries permitted until 2026-11-04 00:57:04.929443819 +0000 UTC m=+6.118611251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xswcp" (UniqueName: "kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp") pod "network-check-target-zh5bh" (UID: "66c16477-f3db-43cf-9830-39fe0267bdce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:02.929622 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.929329 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:02.929622 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:02.929491 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:04.929482591 +0000 UTC m=+6.118650009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:03.240691 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:03.240606 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:52:00 +0000 UTC" deadline="2028-06-12 18:11:35.612302395 +0000 UTC" Nov 04 00:57:03.240691 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:03.240644 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14081h14m32.371661968s" Nov 04 00:57:03.271961 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:03.271257 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:03.271961 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:03.271436 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:03.271961 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:03.271477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:03.271961 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:03.271581 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:03.393495 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:03.393402 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"1a84db2d45c5cb12625222b3483c125f","Type":"ContainerStarted","Data":"3d6f8fa1163a57e94ce746ea37b9da17d2892244215162f8bec361a7959d5c3a"} Nov 04 00:57:03.409272 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:03.408500 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" podStartSLOduration=3.408485114 podStartE2EDuration="3.408485114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:03.408085205 +0000 UTC m=+4.597252639" watchObservedRunningTime="2026-11-04 00:57:03.408485114 +0000 UTC m=+4.597652550" Nov 04 00:57:04.270502 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:04.270151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:04.270502 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.270254 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:04.947020 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:04.946980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:04.947231 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:04.947033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:04.947231 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:04.947071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:04.947356 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.947250 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:04.947356 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.947268 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:04.947356 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.947279 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xswcp for pod openshift-network-diagnostics/network-check-target-zh5bh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:04.947356 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.947333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp podName:66c16477-f3db-43cf-9830-39fe0267bdce nodeName:}" failed. No retries permitted until 2026-11-04 00:57:08.947316821 +0000 UTC m=+10.136484241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xswcp" (UniqueName: "kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp") pod "network-check-target-zh5bh" (UID: "66c16477-f3db-43cf-9830-39fe0267bdce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:04.947579 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.947399 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:04.947579 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.947432 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret podName:04f82b22-3949-44d3-abe9-6473a2f68f76 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:08.947421725 +0000 UTC m=+10.136589144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret") pod "global-pull-secret-syncer-8rwkl" (UID: "04f82b22-3949-44d3-abe9-6473a2f68f76") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:04.947579 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.947503 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:04.947579 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:04.947535 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:08.94752462 +0000 UTC m=+10.136692040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:05.272671 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:05.272565 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:05.273130 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:05.272738 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:05.273351 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:05.273330 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:05.273487 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:05.273463 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:05.405804 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:05.405762 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="0c6c11b4ec6a9ba9445feab09cbe71d6b8a4fcd23ab1d9a4495f1fda87eb21c4" exitCode=0 Nov 04 00:57:05.405941 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:05.405815 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" event={"ID":"0c8ada62-5c11-4462-a29d-a576d7c162df","Type":"ContainerDied","Data":"0c6c11b4ec6a9ba9445feab09cbe71d6b8a4fcd23ab1d9a4495f1fda87eb21c4"} Nov 04 00:57:06.270256 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:06.270220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:06.270404 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:06.270358 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:06.417574 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:06.417537 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w6tt8" event={"ID":"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31","Type":"ContainerStarted","Data":"ea2763b6532e77f08db270d19f75f2a9c48f50b7278d65f7e77d19451c543cb1"} Nov 04 00:57:06.436865 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:06.436793 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w6tt8" podStartSLOduration=3.6725748129999998 podStartE2EDuration="7.436749487s" totalImagesPullingTime="3.764174674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:06.436311718 +0000 UTC m=+7.625479156" watchObservedRunningTime="2026-11-04 00:57:06.436749487 +0000 UTC m=+7.625916925" Nov 04 00:57:07.269729 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:07.269660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:07.269882 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:07.269797 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:07.270201 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:07.270180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:07.270315 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:07.270293 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:07.426692 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:07.426651 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" event={"ID":"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7","Type":"ContainerStarted","Data":"98a640c8c35af0f80a49ef93ebf21730783d9a9ffcbd5b1f3a251260b283f1e8"} Nov 04 00:57:07.431200 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:07.431170 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dcksb" event={"ID":"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56","Type":"ContainerStarted","Data":"14b8e35c330971a96c5834e0ac0e082706495fc758273f88b8b55890ea0150fe"} Nov 04 00:57:08.270036 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:08.269955 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:08.270183 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.270100 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:08.328967 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:08.328935 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:08.329772 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:08.329748 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:08.348591 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:08.348537 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dcksb" podStartSLOduration=4.699287288 podStartE2EDuration="9.348519876s" totalImagesPullingTime="4.649232588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:07.446422375 +0000 UTC m=+8.635589806" watchObservedRunningTime="2026-11-04 00:57:08.348519876 +0000 UTC m=+9.537687314" Nov 04 00:57:08.671518 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:08.671494 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:57:08.980406 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:08.980325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:08.980406 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:08.980382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:08.980603 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:08.980421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:08.980603 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.980575 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:08.980603 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.980592 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:08.980603 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.980606 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xswcp for pod openshift-network-diagnostics/network-check-target-zh5bh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:08.980750 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.980663 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp podName:66c16477-f3db-43cf-9830-39fe0267bdce nodeName:}" failed. No retries permitted until 2026-11-04 00:57:16.980645002 +0000 UTC m=+18.169812434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xswcp" (UniqueName: "kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp") pod "network-check-target-zh5bh" (UID: "66c16477-f3db-43cf-9830-39fe0267bdce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:08.981090 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.981064 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:08.981198 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.981134 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret podName:04f82b22-3949-44d3-abe9-6473a2f68f76 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:16.981103818 +0000 UTC m=+18.170271248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret") pod "global-pull-secret-syncer-8rwkl" (UID: "04f82b22-3949-44d3-abe9-6473a2f68f76") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:08.981198 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.981191 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:08.981311 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:08.981223 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:16.981211562 +0000 UTC m=+18.170378991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:09.265622 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.264912 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:57:08.671516682Z","UUID":"3508cf9c-052c-49cd-8cf1-b10b7954384e","Handler":null,"Name":"","Endpoint":""} Nov 04 00:57:09.266557 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.266536 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:57:09.266679 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.266565 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:57:09.273819 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.273477 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:09.273819 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:09.273604 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:09.273819 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.273657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:09.273819 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:09.273771 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:09.441045 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.441008 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dm58x" event={"ID":"08221c6a-1a73-4922-a769-caeb42c194bb","Type":"ContainerStarted","Data":"89bfa760104453f0cb71a46da739fd748d42dca0f91d151f899d8f2b9247926a"} Nov 04 00:57:09.444677 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.444655 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:09.444802 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.444693 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" event={"ID":"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7","Type":"ContainerStarted","Data":"352a2dccb5caf7ca8d98fd2363b76fe1f8dc7bfab75c8904a438503b1d03d1d2"} Nov 04 00:57:09.460058 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:09.459998 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dm58x" podStartSLOduration=3.9762598110000003 podStartE2EDuration="11.459982557s" totalImagesPullingTime="7.483722746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:09.459848072 +0000 UTC m=+10.649015511" watchObservedRunningTime="2026-11-04 00:57:09.459982557 +0000 UTC m=+10.649149995" Nov 04 00:57:10.269768 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:10.269584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:10.270222 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:10.269859 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:10.449621 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:10.449538 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" event={"ID":"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7","Type":"ContainerStarted","Data":"8a1092cb85563b631c3edc06d522e5eab73d3ccf211521d4b2e4ad407bfc31a7"} Nov 04 00:57:10.452392 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:10.452309 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="cbedca9ef1feebccc002dbaf4d819756a55225e290ab7a2cd50ecc71799ced10" exitCode=0 Nov 04 00:57:10.452577 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:10.452559 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" event={"ID":"0c8ada62-5c11-4462-a29d-a576d7c162df","Type":"ContainerDied","Data":"cbedca9ef1feebccc002dbaf4d819756a55225e290ab7a2cd50ecc71799ced10"} Nov 04 00:57:10.455068 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:10.455023 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-shjsf" event={"ID":"51ff9909-e56c-4d27-be31-574bf33a8f75","Type":"ContainerStarted","Data":"45a4fda290bee1b969c07436bdfbbd412a7a90293ce2530705d6fe11991171fd"} Nov 04 00:57:10.467967 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:10.467921 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" podStartSLOduration=3.318385023 podStartE2EDuration="11.467906548s" totalImagesPullingTime="8.149521525s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:10.467228722 +0000 UTC m=+11.656396160" watchObservedRunningTime="2026-11-04 00:57:10.467906548 +0000 UTC m=+11.657073987" Nov 04 00:57:10.481896 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:10.481857 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-shjsf" podStartSLOduration=3.698543082 podStartE2EDuration="11.481846023s" totalImagesPullingTime="7.783302941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:10.481458109 +0000 UTC m=+11.670625547" watchObservedRunningTime="2026-11-04 00:57:10.481846023 +0000 UTC m=+11.671013461" Nov 04 00:57:11.269626 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:11.269598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:11.269798 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:11.269642 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:11.269798 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:11.269757 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:11.270368 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:11.269876 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:11.458998 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:11.458967 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" event={"ID":"51fbfff7-c90b-4726-bbff-6ebbcc943626","Type":"ContainerStarted","Data":"0e2586d6762d7e02d777032dfbb23fe04fb6bbe43e33e9f162bff22283220b81"} Nov 04 00:57:11.461042 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:11.460991 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fkvdp" event={"ID":"3391f8ba-2fe6-4950-b644-5361fc8c2e48","Type":"ContainerStarted","Data":"0ced86d2dab442c732f4b6badbad53b244ffa834e898d58f88c632ab9d51349b"} Nov 04 00:57:11.475082 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:11.475033 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" podStartSLOduration=2.7122783950000002 podStartE2EDuration="12.475017844s" totalImagesPullingTime="9.762739449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:11.474817341 +0000 UTC m=+12.663984779" watchObservedRunningTime="2026-11-04 00:57:11.475017844 +0000 UTC m=+12.664185289" Nov 04 00:57:11.490053 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:11.490006 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fkvdp" podStartSLOduration=5.94244179 podStartE2EDuration="13.48999335s" totalImagesPullingTime="7.54755156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:11.489670257 +0000 UTC m=+12.678837695" watchObservedRunningTime="2026-11-04 00:57:11.48999335 +0000 UTC m=+12.679160787" Nov 04 00:57:12.269693 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:12.269662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:12.269851 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:12.269780 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:12.466303 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:12.466255 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="e62ba671b9e105de2d8dd37a0a87a364a50379722ab93dc37cb62bf93189cd17" exitCode=0 Nov 04 00:57:12.466452 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:12.466356 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" event={"ID":"0c8ada62-5c11-4462-a29d-a576d7c162df","Type":"ContainerDied","Data":"e62ba671b9e105de2d8dd37a0a87a364a50379722ab93dc37cb62bf93189cd17"} Nov 04 00:57:13.270064 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:13.270029 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:13.270462 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:13.270161 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:13.270522 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:13.270507 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:13.270593 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:13.270579 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:14.270568 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:14.270386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:14.270975 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:14.270667 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:14.473726 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:14.473689 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="b036c256f474e00da915d93ffcdf96bb7b1da4019e3e9d3a8c8f8a1b6428e49a" exitCode=0 Nov 04 00:57:14.473881 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:14.473731 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" event={"ID":"0c8ada62-5c11-4462-a29d-a576d7c162df","Type":"ContainerDied","Data":"b036c256f474e00da915d93ffcdf96bb7b1da4019e3e9d3a8c8f8a1b6428e49a"} Nov 04 00:57:15.270425 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:15.269909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:15.270425 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:15.270063 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:15.270425 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:15.270102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:15.270425 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:15.270248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:16.270601 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:16.270415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:16.270926 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:16.270688 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:16.481747 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:16.481663 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"f72f1f5eed20245759ce50cb0b03ef7dacf72a1e538c6a775af79ca19d44582a"} Nov 04 00:57:16.481895 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:16.481750 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"ec69bf925dcbebfcd0ad0bfe1650394454091fec97a8aa528d89865e3d4ba95b"} Nov 04 00:57:16.481895 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:16.481794 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"16569d881af975159a71657a943e31c09380cab59ae81f198334ecc8b6c71363"} Nov 04 00:57:16.481895 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:16.481833 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"6f54f292fa6697cbd9ea88cc2eab7fdbd77ba3e4b7b77fa4f34aaf5aa0c31ff8"} Nov 04 00:57:16.481895 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:16.481872 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"b90d60644c5b7909708e4ce8007c70538667383ba9a092ef242977a400d707f0"} Nov 04 00:57:16.482042 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:16.481916 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"9e8c9ac236f0c6138cdee1dcdb0000343af8b8b3d3883306f950ef522c59e262"} Nov 04 00:57:17.041700 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:17.041624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:17.041700 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:17.041672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:17.041700 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:17.041701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:17.041952 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.041787 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:17.041952 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.041795 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:17.041952 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.041832 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:33.041820783 +0000 UTC m=+34.230988211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 00:57:17.041952 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.041854 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret podName:04f82b22-3949-44d3-abe9-6473a2f68f76 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:33.041838573 +0000 UTC m=+34.231005988 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret") pod "global-pull-secret-syncer-8rwkl" (UID: "04f82b22-3949-44d3-abe9-6473a2f68f76") : object "kube-system"/"original-pull-secret" not registered Nov 04 00:57:17.041952 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.041790 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 00:57:17.041952 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.041882 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 00:57:17.041952 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.041894 2578 projected.go:196] Error preparing data for projected volume kube-api-access-xswcp for pod openshift-network-diagnostics/network-check-target-zh5bh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:17.041952 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.041937 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp podName:66c16477-f3db-43cf-9830-39fe0267bdce nodeName:}" failed. No retries permitted until 2026-11-04 00:57:33.041923474 +0000 UTC m=+34.231090893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xswcp" (UniqueName: "kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp") pod "network-check-target-zh5bh" (UID: "66c16477-f3db-43cf-9830-39fe0267bdce") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 00:57:17.269793 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:17.269765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:17.269793 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:17.269786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:17.269991 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.269865 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:17.270029 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:17.270002 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:17.486001 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:17.485968 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="ea3003a9de0b30293c93928e29dfd73b997c3966703033d2a227cb5caf5c6b3e" exitCode=0 Nov 04 00:57:17.486490 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:17.486020 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" event={"ID":"0c8ada62-5c11-4462-a29d-a576d7c162df","Type":"ContainerDied","Data":"ea3003a9de0b30293c93928e29dfd73b997c3966703033d2a227cb5caf5c6b3e"} Nov 04 00:57:18.269961 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:18.269932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:18.270098 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:18.270019 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:18.492197 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:18.492158 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"aa9f44432031626d523c0e6eec0fefe23f855ffbe7292a105325382e4bf485db"} Nov 04 00:57:18.495419 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:18.495385 2578 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="afa41aa91dba70cd6b5650ec75317bae7e12811f5876d6809a2985e218368365" exitCode=0 Nov 04 00:57:18.495532 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:18.495434 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" event={"ID":"0c8ada62-5c11-4462-a29d-a576d7c162df","Type":"ContainerDied","Data":"afa41aa91dba70cd6b5650ec75317bae7e12811f5876d6809a2985e218368365"} Nov 04 00:57:19.270761 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:19.270728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:19.270944 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:19.270853 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:19.270944 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:19.270899 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:19.271055 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:19.270983 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:19.500624 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:19.500590 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" event={"ID":"0c8ada62-5c11-4462-a29d-a576d7c162df","Type":"ContainerStarted","Data":"dd450d488f00af46b565b83118098e85efdb5a2f01a54629c2e299c7debbaa85"} Nov 04 00:57:19.523793 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:19.523659 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rlwsw" podStartSLOduration=10.093261436 podStartE2EDuration="21.523640548s" totalImagesPullingTime="11.430379112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:19.521068715 +0000 UTC m=+20.710236151" watchObservedRunningTime="2026-11-04 00:57:19.523640548 +0000 UTC m=+20.712807986" Nov 04 00:57:20.269833 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:20.269790 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:20.270016 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:20.269897 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:21.269912 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:21.269727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:21.270473 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:21.269785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:21.270473 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:21.269980 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:21.270473 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:21.270049 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:21.507991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:21.507958 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" event={"ID":"1006149a-92ee-43c1-bb41-6aaf1df41c36","Type":"ContainerStarted","Data":"93602cd049b4950790b73015c7a32ee39653459057cdff2a8d77c73ebf6f47ff"} Nov 04 00:57:21.508265 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:21.508242 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:21.524940 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:21.524879 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:21.535435 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:21.535395 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" podStartSLOduration=8.134959076 podStartE2EDuration="22.535382638s" totalImagesPullingTime="14.400423562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:21.533771548 +0000 UTC m=+22.722938984" watchObservedRunningTime="2026-11-04 00:57:21.535382638 +0000 UTC m=+22.724550074" Nov 04 00:57:22.226976 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.226945 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8rwkl"] Nov 04 00:57:22.227157 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.227063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:22.227208 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:22.227175 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:22.230887 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.230856 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zh5bh"] Nov 04 00:57:22.230998 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.230950 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m84bl"] Nov 04 00:57:22.231069 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.231057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:22.231239 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:22.231217 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:22.231293 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.231259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:22.231501 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:22.231465 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:22.510487 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.510446 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:22.510879 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.510529 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:22.526993 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:22.525706 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:24.269764 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:24.269738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:24.270221 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:24.269739 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:24.270221 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:24.269836 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:24.270221 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:24.269925 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:24.270221 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:24.269740 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:24.270221 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:24.269998 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:26.269524 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:26.269489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:26.270090 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:26.269490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:26.270090 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:26.269604 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:26.270090 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:26.269490 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:26.270090 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:26.269673 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:26.270090 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:26.269763 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:28.269856 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.269825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:28.270349 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.269821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:28.270349 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.269947 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rwkl" podUID="04f82b22-3949-44d3-abe9-6473a2f68f76" Nov 04 00:57:28.270349 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.270012 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m84bl" podUID="5b2c9231-5813-4143-93ca-2457278f37a2" Nov 04 00:57:28.270349 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.269840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:28.270349 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.270075 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zh5bh" podUID="66c16477-f3db-43cf-9830-39fe0267bdce" Nov 04 00:57:28.610336 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.610302 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeReady" Nov 04 00:57:28.610503 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.610448 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 00:57:28.646986 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.646954 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wmhkb"] Nov 04 00:57:28.651440 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.651394 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bw7wx"] Nov 04 00:57:28.651573 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.651526 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" Nov 04 00:57:28.654683 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.654661 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6"] Nov 04 00:57:28.654779 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.654763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" Nov 04 00:57:28.657543 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.657518 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hwnjk"] Nov 04 00:57:28.658378 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.658356 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.661185 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.661158 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mh5d6"] Nov 04 00:57:28.661474 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.661447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" Nov 04 00:57:28.665271 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.665059 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4"] Nov 04 00:57:28.665271 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.665194 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.665271 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.665197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:28.665271 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.665229 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.671045 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.671011 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sgnvg"] Nov 04 00:57:28.671611 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.671555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.674929 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.674672 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75c85b76dc-nfpkl"] Nov 04 00:57:28.675143 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.675081 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.678220 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.675990 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7dq87\"" Nov 04 00:57:28.678220 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.676113 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fjgn7\"" Nov 04 00:57:28.678220 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.676199 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 00:57:28.678220 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.676249 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 00:57:28.681232 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.681209 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn"] Nov 04 00:57:28.681469 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.681452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.688179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.688158 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c"] Nov 04 00:57:28.689394 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.689244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.690710 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.690692 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8"] Nov 04 00:57:28.691075 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.691058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" Nov 04 00:57:28.693973 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.693954 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bfcf87bc8-zhslb"] Nov 04 00:57:28.694066 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.694048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.699950 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.699930 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-669749fcf7-vqftg"] Nov 04 00:57:28.700067 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.700050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.702493 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.702474 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5"] Nov 04 00:57:28.702613 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.702595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.706970 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.706953 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7vtl7"] Nov 04 00:57:28.707067 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.707053 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" Nov 04 00:57:28.711749 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.711707 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wmhkb"] Nov 04 00:57:28.711749 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.711747 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mh5d6"] Nov 04 00:57:28.712014 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.711772 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mvtzk"] Nov 04 00:57:28.712191 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.712175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.715142 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.715105 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vl6w6"] Nov 04 00:57:28.715241 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.715221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.718423 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718403 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hwnjk"] Nov 04 00:57:28.718501 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718438 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c"] Nov 04 00:57:28.718501 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718461 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8"] Nov 04 00:57:28.718501 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718481 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5"] Nov 04 00:57:28.718501 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718500 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn"] Nov 04 00:57:28.718640 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:28.718640 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718522 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6"] Nov 04 00:57:28.718640 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718543 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sgnvg"] Nov 04 00:57:28.718640 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718562 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-669749fcf7-vqftg"] Nov 04 00:57:28.718640 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718581 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bw7wx"] Nov 04 00:57:28.718640 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718598 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bfcf87bc8-zhslb"] Nov 04 00:57:28.718640 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718621 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7vtl7"] Nov 04 00:57:28.718883 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718646 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4"] Nov 04 00:57:28.718883 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718666 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mvtzk"] Nov 04 00:57:28.718883 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718692 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75c85b76dc-nfpkl"] Nov 04 00:57:28.718883 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.718716 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vl6w6"] Nov 04 00:57:28.726195 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/261f5fa6-5956-4a5d-a104-982dc13aed39-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:28.726440 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-service-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.726440 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d648b600-369c-47b0-ba02-6b3df1252bed-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.726440 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98tj\" (UniqueName: \"kubernetes.io/projected/d648b600-369c-47b0-ba02-6b3df1252bed-kube-api-access-n98tj\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.726440 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlnpx\" (UniqueName: \"kubernetes.io/projected/ede327f8-e1cf-4596-abe5-de379f528b15-kube-api-access-xlnpx\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.726440 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jddfl\" (UniqueName: \"kubernetes.io/projected/f4bef706-ca6f-4260-962f-ad0599ad093a-kube-api-access-jddfl\") pod \"kas-connection-checker-685895c7c9-bw7wx\" (UID: \"f4bef706-ca6f-4260-962f-ad0599ad093a\") " pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" Nov 04 00:57:28.726936 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.726936 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726612 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:28.726936 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.726936 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726711 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.726936 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5r4\" (UniqueName: \"kubernetes.io/projected/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-kube-api-access-jv5r4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.727186 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.726981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskr9\" (UniqueName: \"kubernetes.io/projected/767984a4-597e-4498-a6bf-c988a7b228d1-kube-api-access-gskr9\") pod \"kas-connection-checker-685895c7c9-wmhkb\" (UID: \"767984a4-597e-4498-a6bf-c988a7b228d1\") " pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" Nov 04 00:57:28.727186 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.727008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-snapshots\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.727186 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.727027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ede327f8-e1cf-4596-abe5-de379f528b15-serving-cert\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.727186 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.727045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.727186 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.727063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-tmp\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.727186 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.727081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7l6w\" (UniqueName: \"kubernetes.io/projected/ac81ee2f-5c6a-4ead-89c5-2a181b928494-kube-api-access-x7l6w\") pod \"kas-connection-checker-685895c7c9-hwnjk\" (UID: \"ac81ee2f-5c6a-4ead-89c5-2a181b928494\") " pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" Nov 04 00:57:28.750478 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.750448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-w6tt8_6ef08de9-97a2-4c5b-ae3a-66a9cd783a31/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:28.763737 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.763715 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.763865 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.763767 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 00:57:28.763865 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.763860 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.763979 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.763887 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.763979 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.763902 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.763979 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.763949 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.763979 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.763947 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 00:57:28.764179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764036 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.764179 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764062 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 00:57:28.764296 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764210 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 00:57:28.764296 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764209 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 00:57:28.764296 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764242 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.764296 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764260 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.764296 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764243 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.764483 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764316 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:57:28.764483 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764350 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.764483 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764354 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.764483 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764386 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.764483 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764427 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.764624 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764492 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.764624 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764542 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.764624 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764601 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.764624 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764608 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.764734 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764648 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.764764 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764737 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 00:57:28.764808 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.764788 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:28.765826 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.765807 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 00:57:28.765893 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.765850 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:57:28.775155 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775112 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 00:57:28.775284 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775178 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 00:57:28.775284 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775215 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 00:57:28.775284 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775217 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 00:57:28.775650 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775618 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 00:57:28.775739 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775656 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 00:57:28.775739 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775669 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 00:57:28.775739 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775715 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 00:57:28.775932 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775916 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 00:57:28.776072 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775933 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 00:57:28.776072 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775946 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:57:28.776072 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775971 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:57:28.776072 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775978 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 00:57:28.776072 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.775984 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 00:57:28.776326 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776248 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gzs44\"" Nov 04 00:57:28.776326 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776278 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 00:57:28.776326 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776295 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6kp4v\"" Nov 04 00:57:28.776326 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776306 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 00:57:28.776326 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776321 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d6bjj\"" Nov 04 00:57:28.776493 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776325 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9t5w2\"" Nov 04 00:57:28.776493 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776366 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mn9vp\"" Nov 04 00:57:28.776493 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776383 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bcswz\"" Nov 04 00:57:28.776493 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776428 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cjss9\"" Nov 04 00:57:28.776493 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776431 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mpjzr\"" Nov 04 00:57:28.776493 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776464 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7rnfb\"" Nov 04 00:57:28.776493 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776484 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rppzs\"" Nov 04 00:57:28.776801 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776520 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 00:57:28.776801 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.776598 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pb69q\"" Nov 04 00:57:28.827532 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ede327f8-e1cf-4596-abe5-de379f528b15-serving-cert\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.827532 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827535 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.827770 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.827770 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-tmp\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.827770 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7l6w\" (UniqueName: \"kubernetes.io/projected/ac81ee2f-5c6a-4ead-89c5-2a181b928494-kube-api-access-x7l6w\") pod \"kas-connection-checker-685895c7c9-hwnjk\" (UID: \"ac81ee2f-5c6a-4ead-89c5-2a181b928494\") " pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" Nov 04 00:57:28.827770 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.827770 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.828050 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.828050 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.828050 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d648b600-369c-47b0-ba02-6b3df1252bed-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.828050 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.827967 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.828050 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828040 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-tmp\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.828313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/261f5fa6-5956-4a5d-a104-982dc13aed39-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:28.828313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-service-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.828313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-stats-auth\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.828313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dqxb\" (UniqueName: \"kubernetes.io/projected/b0807c2d-b756-42e8-86b3-97c30f23df72-kube-api-access-5dqxb\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.828313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72nkw\" (UniqueName: \"kubernetes.io/projected/7d024224-166d-4cb4-976f-75e0b47addae-kube-api-access-72nkw\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.828313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.828545 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc462\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.828545 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7xnz\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.828545 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:28.828545 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbd7a56-ab21-413c-abac-36b1f79f452e-cert\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:28.828699 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rs4s\" (UniqueName: \"kubernetes.io/projected/969ca7d8-004d-4f22-9929-96099b5f011c-kube-api-access-5rs4s\") pod \"network-check-source-6f78d59f99-xqsk5\" (UID: \"969ca7d8-004d-4f22-9929-96099b5f011c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" Nov 04 00:57:28.828699 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.828699 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.828828 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.828705 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:57:28.828828 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/261f5fa6-5956-4a5d-a104-982dc13aed39-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:28.828828 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.828771 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert podName:261f5fa6-5956-4a5d-a104-982dc13aed39 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.328754449 +0000 UTC m=+30.517921871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mh5d6" (UID: "261f5fa6-5956-4a5d-a104-982dc13aed39") : secret "networking-console-plugin-cert" not found Nov 04 00:57:28.828828 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-service-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.828828 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828801 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d648b600-369c-47b0-ba02-6b3df1252bed-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.829043 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.829043 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.829043 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828934 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76kwx\" (UniqueName: \"kubernetes.io/projected/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-kube-api-access-76kwx\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.829043 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.828973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5r4\" (UniqueName: \"kubernetes.io/projected/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-kube-api-access-jv5r4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.829043 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskr9\" (UniqueName: \"kubernetes.io/projected/767984a4-597e-4498-a6bf-c988a7b228d1-kube-api-access-gskr9\") pod \"kas-connection-checker-685895c7c9-wmhkb\" (UID: \"767984a4-597e-4498-a6bf-c988a7b228d1\") " pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" Nov 04 00:57:28.829043 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9x4t\" (UniqueName: \"kubernetes.io/projected/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd-kube-api-access-d9x4t\") pod \"volume-data-source-validator-58c9f5f46d-pv57c\" (UID: \"e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" Nov 04 00:57:28.829339 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-snapshots\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.829339 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0807c2d-b756-42e8-86b3-97c30f23df72-config-volume\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.829339 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aa9b05c-8730-4590-8709-0275e8be2631-config\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.829339 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98tj\" (UniqueName: \"kubernetes.io/projected/d648b600-369c-47b0-ba02-6b3df1252bed-kube-api-access-n98tj\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.829339 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlnpx\" (UniqueName: \"kubernetes.io/projected/ede327f8-e1cf-4596-abe5-de379f528b15-kube-api-access-xlnpx\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.829339 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddfl\" (UniqueName: \"kubernetes.io/projected/f4bef706-ca6f-4260-962f-ad0599ad093a-kube-api-access-jddfl\") pod \"kas-connection-checker-685895c7c9-bw7wx\" (UID: \"f4bef706-ca6f-4260-962f-ad0599ad093a\") " pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" Nov 04 00:57:28.829339 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-default-certificate\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.829339 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.829673 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.829673 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.829673 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-config\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.829673 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-snapshots\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-serving-cert\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.829782 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aa9b05c-8730-4590-8709-0275e8be2631-serving-cert\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.829884 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830255 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830312 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0807c2d-b756-42e8-86b3-97c30f23df72-tmp-dir\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.830462 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830456 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.831084 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29mnh\" (UniqueName: \"kubernetes.io/projected/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-kube-api-access-29mnh\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.831084 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830496 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.831084 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvpsh\" (UniqueName: \"kubernetes.io/projected/8dbd7a56-ab21-413c-abac-36b1f79f452e-kube-api-access-tvpsh\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:28.831084 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.830570 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls podName:d648b600-369c-47b0-ba02-6b3df1252bed nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.330557789 +0000 UTC m=+30.519725204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kxzp6" (UID: "d648b600-369c-47b0-ba02-6b3df1252bed") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:28.831084 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.831084 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.831084 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.830668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzr6\" (UniqueName: \"kubernetes.io/projected/6aa9b05c-8730-4590-8709-0275e8be2631-kube-api-access-nwzr6\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.832688 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.832668 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ede327f8-e1cf-4596-abe5-de379f528b15-serving-cert\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.832688 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.832679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.835879 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.835852 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7l6w\" (UniqueName: \"kubernetes.io/projected/ac81ee2f-5c6a-4ead-89c5-2a181b928494-kube-api-access-x7l6w\") pod \"kas-connection-checker-685895c7c9-hwnjk\" (UID: \"ac81ee2f-5c6a-4ead-89c5-2a181b928494\") " pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" Nov 04 00:57:28.839520 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.839491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5r4\" (UniqueName: \"kubernetes.io/projected/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-kube-api-access-jv5r4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:28.840051 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.840028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskr9\" (UniqueName: \"kubernetes.io/projected/767984a4-597e-4498-a6bf-c988a7b228d1-kube-api-access-gskr9\") pod \"kas-connection-checker-685895c7c9-wmhkb\" (UID: \"767984a4-597e-4498-a6bf-c988a7b228d1\") " pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" Nov 04 00:57:28.840331 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.840314 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlnpx\" (UniqueName: \"kubernetes.io/projected/ede327f8-e1cf-4596-abe5-de379f528b15-kube-api-access-xlnpx\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:28.840590 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.840572 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98tj\" (UniqueName: \"kubernetes.io/projected/d648b600-369c-47b0-ba02-6b3df1252bed-kube-api-access-n98tj\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:28.840671 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.840651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddfl\" (UniqueName: \"kubernetes.io/projected/f4bef706-ca6f-4260-962f-ad0599ad093a-kube-api-access-jddfl\") pod \"kas-connection-checker-685895c7c9-bw7wx\" (UID: \"f4bef706-ca6f-4260-962f-ad0599ad093a\") " pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" Nov 04 00:57:28.931720 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.931687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.931878 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.931729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.931878 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.931749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-stats-auth\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.931878 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.931765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqxb\" (UniqueName: \"kubernetes.io/projected/b0807c2d-b756-42e8-86b3-97c30f23df72-kube-api-access-5dqxb\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.931878 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.931781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72nkw\" (UniqueName: \"kubernetes.io/projected/7d024224-166d-4cb4-976f-75e0b47addae-kube-api-access-72nkw\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.931878 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.931803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.931878 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.931821 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc462\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.931878 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.931839 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.931906 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls podName:8cf9dde4-e870-49c8-a34d-1c0a4c85acb6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.431886435 +0000 UTC m=+30.621053863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69rhn" (UID: "8cf9dde4-e870-49c8-a34d-1c0a4c85acb6") : secret "samples-operator-tls" not found Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.931849 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7xnz\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbd7a56-ab21-413c-abac-36b1f79f452e-cert\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rs4s\" (UniqueName: \"kubernetes.io/projected/969ca7d8-004d-4f22-9929-96099b5f011c-kube-api-access-5rs4s\") pod \"network-check-source-6f78d59f99-xqsk5\" (UID: \"969ca7d8-004d-4f22-9929-96099b5f011c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.932273 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kwx\" (UniqueName: \"kubernetes.io/projected/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-kube-api-access-76kwx\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932279 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932284 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9x4t\" (UniqueName: \"kubernetes.io/projected/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd-kube-api-access-d9x4t\") pod \"volume-data-source-validator-58c9f5f46d-pv57c\" (UID: \"e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0807c2d-b756-42e8-86b3-97c30f23df72-config-volume\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aa9b05c-8730-4590-8709-0275e8be2631-config\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932388 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-default-certificate\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-config\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-serving-cert\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aa9b05c-8730-4590-8709-0275e8be2631-serving-cert\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0807c2d-b756-42e8-86b3-97c30f23df72-tmp-dir\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.932686 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932690 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.932730 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.932744 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bfcf87bc8-zhslb: secret "image-registry-tls" not found Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29mnh\" (UniqueName: \"kubernetes.io/projected/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-kube-api-access-29mnh\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvpsh\" (UniqueName: \"kubernetes.io/projected/8dbd7a56-ab21-413c-abac-36b1f79f452e-kube-api-access-tvpsh\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.932804 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls podName:e54204af-fcd5-442e-9009-f58ba803f3c4 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.432787448 +0000 UTC m=+30.621954946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls") pod "image-registry-6bfcf87bc8-zhslb" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4") : secret "image-registry-tls" not found Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzr6\" (UniqueName: \"kubernetes.io/projected/6aa9b05c-8730-4590-8709-0275e8be2631-kube-api-access-nwzr6\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.932975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.933001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.933028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.933517 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.933289 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.934258 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.933551 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:57:28.934258 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.933602 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.433587443 +0000 UTC m=+30.622754878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : secret "router-metrics-certs-default" not found Nov 04 00:57:28.934258 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.933614 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.934258 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.933701 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:28.934258 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.933715 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c85b76dc-nfpkl: secret "image-registry-tls" not found Nov 04 00:57:28.934258 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.933762 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls podName:3603f9b4-53ae-4e45-832f-e5c29ca743b9 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.433749178 +0000 UTC m=+30.622916612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls") pod "image-registry-75c85b76dc-nfpkl" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9") : secret "image-registry-tls" not found Nov 04 00:57:28.934258 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.933950 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-config\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.934258 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.934073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0807c2d-b756-42e8-86b3-97c30f23df72-tmp-dir\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.934629 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.934539 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.935278 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.934929 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0807c2d-b756-42e8-86b3-97c30f23df72-config-volume\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.935278 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.934969 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.935278 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.934998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.935278 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.935076 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:28.935278 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.935089 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.435072325 +0000 UTC m=+30.624239772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:28.935278 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:28.935150 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls podName:b0807c2d-b756-42e8-86b3-97c30f23df72 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:29.435109159 +0000 UTC m=+30.624276580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls") pod "dns-default-mvtzk" (UID: "b0807c2d-b756-42e8-86b3-97c30f23df72") : secret "dns-default-metrics-tls" not found Nov 04 00:57:28.935655 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.935584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aa9b05c-8730-4590-8709-0275e8be2631-config\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.936189 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.936144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.936793 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.936767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-stats-auth\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.937056 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.937035 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aa9b05c-8730-4590-8709-0275e8be2631-serving-cert\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.937227 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.937199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.937556 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.937540 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-default-certificate\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.937836 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.937819 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-serving-cert\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.938250 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.938225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.938622 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.938605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbd7a56-ab21-413c-abac-36b1f79f452e-cert\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:28.941865 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.941843 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc462\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.942017 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.941989 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rs4s\" (UniqueName: \"kubernetes.io/projected/969ca7d8-004d-4f22-9929-96099b5f011c-kube-api-access-5rs4s\") pod \"network-check-source-6f78d59f99-xqsk5\" (UID: \"969ca7d8-004d-4f22-9929-96099b5f011c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" Nov 04 00:57:28.943771 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.943470 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72nkw\" (UniqueName: \"kubernetes.io/projected/7d024224-166d-4cb4-976f-75e0b47addae-kube-api-access-72nkw\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:28.943771 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.943585 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqxb\" (UniqueName: \"kubernetes.io/projected/b0807c2d-b756-42e8-86b3-97c30f23df72-kube-api-access-5dqxb\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:28.944326 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.944303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7xnz\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.946325 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.946280 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kwx\" (UniqueName: \"kubernetes.io/projected/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-kube-api-access-76kwx\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:28.946418 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.946341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzr6\" (UniqueName: \"kubernetes.io/projected/6aa9b05c-8730-4590-8709-0275e8be2631-kube-api-access-nwzr6\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:28.946418 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.946392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvpsh\" (UniqueName: \"kubernetes.io/projected/8dbd7a56-ab21-413c-abac-36b1f79f452e-kube-api-access-tvpsh\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:28.946418 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.946400 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29mnh\" (UniqueName: \"kubernetes.io/projected/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-kube-api-access-29mnh\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:28.947564 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.947548 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:28.947827 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.947805 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9x4t\" (UniqueName: \"kubernetes.io/projected/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd-kube-api-access-d9x4t\") pod \"volume-data-source-validator-58c9f5f46d-pv57c\" (UID: \"e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" Nov 04 00:57:28.948281 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.948261 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:28.952903 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.952877 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-w6tt8_6ef08de9-97a2-4c5b-ae3a-66a9cd783a31/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:28.968288 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.968264 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" Nov 04 00:57:28.991607 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.991580 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" Nov 04 00:57:28.998901 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:28.998844 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" Nov 04 00:57:29.021752 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.021718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:29.035775 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.032175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:29.052107 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.051978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" Nov 04 00:57:29.056046 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.053955 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:29.073197 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.072088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" Nov 04 00:57:29.083313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.082903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:29.094543 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.094365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:29.235766 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.235307 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:29.235766 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.235485 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:29.236094 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.235946 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:29.259049 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.259029 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wmhkb"] Nov 04 00:57:29.261499 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:29.261476 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767984a4_597e_4498_a6bf_c988a7b228d1.slice/crio-f688ab99c75d6ab87c4e41c21a5ea21bbe92b9f815cf6d575a7e673601e1f7ae WatchSource:0}: Error finding container f688ab99c75d6ab87c4e41c21a5ea21bbe92b9f815cf6d575a7e673601e1f7ae: Status 404 returned error can't find the container with id f688ab99c75d6ab87c4e41c21a5ea21bbe92b9f815cf6d575a7e673601e1f7ae Nov 04 00:57:29.281513 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.281488 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bw7wx"] Nov 04 00:57:29.284441 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:29.284411 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bef706_ca6f_4260_962f_ad0599ad093a.slice/crio-920363e7ada1def36659576aa31abf5fc2cbcef52f5bd8a5eaf6dc80ba6371d2 WatchSource:0}: Error finding container 920363e7ada1def36659576aa31abf5fc2cbcef52f5bd8a5eaf6dc80ba6371d2: Status 404 returned error can't find the container with id 920363e7ada1def36659576aa31abf5fc2cbcef52f5bd8a5eaf6dc80ba6371d2 Nov 04 00:57:29.304805 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.304785 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hwnjk"] Nov 04 00:57:29.307760 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:29.307728 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac81ee2f_5c6a_4ead_89c5_2a181b928494.slice/crio-afdc3c7a78e87a2dfa2d67a2961090924b7afe530fe9ecc441f15e941397a578 WatchSource:0}: Error finding container afdc3c7a78e87a2dfa2d67a2961090924b7afe530fe9ecc441f15e941397a578: Status 404 returned error can't find the container with id afdc3c7a78e87a2dfa2d67a2961090924b7afe530fe9ecc441f15e941397a578 Nov 04 00:57:29.338971 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.338932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:29.339168 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.339105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:29.339366 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.339346 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:29.339435 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.339421 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls podName:d648b600-369c-47b0-ba02-6b3df1252bed nodeName:}" failed. No retries permitted until 2026-11-04 00:57:30.339401302 +0000 UTC m=+31.528568734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kxzp6" (UID: "d648b600-369c-47b0-ba02-6b3df1252bed") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:29.339865 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.339843 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:57:29.339967 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.339898 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert podName:261f5fa6-5956-4a5d-a104-982dc13aed39 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:30.33988697 +0000 UTC m=+31.529054386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mh5d6" (UID: "261f5fa6-5956-4a5d-a104-982dc13aed39") : secret "networking-console-plugin-cert" not found Nov 04 00:57:29.353458 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.353429 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4"] Nov 04 00:57:29.380576 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.380552 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-sgnvg"] Nov 04 00:57:29.382958 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.382835 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-w6tt8_6ef08de9-97a2-4c5b-ae3a-66a9cd783a31/konnectivity-agent/0.log" limit=1024 Nov 04 00:57:29.385970 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:29.385939 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede327f8_e1cf_4596_abe5_de379f528b15.slice/crio-5311f6e8245b83b66782847327555239f801dec8e83745900c465a9ddb43afef WatchSource:0}: Error finding container 5311f6e8245b83b66782847327555239f801dec8e83745900c465a9ddb43afef: Status 404 returned error can't find the container with id 5311f6e8245b83b66782847327555239f801dec8e83745900c465a9ddb43afef Nov 04 00:57:29.405770 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.405743 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c"] Nov 04 00:57:29.407622 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.407601 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8"] Nov 04 00:57:29.425768 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.425747 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-7vtl7"] Nov 04 00:57:29.429152 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:29.429107 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9747f7d7_39f9_452d_9e9a_dbc9de48c89e.slice/crio-241c72194a633d194bd11f6c7701f43879c5912719b72bd31b9c38201f356830 WatchSource:0}: Error finding container 241c72194a633d194bd11f6c7701f43879c5912719b72bd31b9c38201f356830: Status 404 returned error can't find the container with id 241c72194a633d194bd11f6c7701f43879c5912719b72bd31b9c38201f356830 Nov 04 00:57:29.439930 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.439900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:29.440028 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.439958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:29.440028 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.439997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:29.440191 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440071 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:29.440191 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.440166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:29.440273 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440201 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls podName:b0807c2d-b756-42e8-86b3-97c30f23df72 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:30.440177553 +0000 UTC m=+31.629344981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls") pod "dns-default-mvtzk" (UID: "b0807c2d-b756-42e8-86b3-97c30f23df72") : secret "dns-default-metrics-tls" not found Nov 04 00:57:29.440273 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440260 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:29.440273 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440273 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bfcf87bc8-zhslb: secret "image-registry-tls" not found Nov 04 00:57:29.440420 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440316 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls podName:e54204af-fcd5-442e-9009-f58ba803f3c4 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:30.440301855 +0000 UTC m=+31.629469285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls") pod "image-registry-6bfcf87bc8-zhslb" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4") : secret "image-registry-tls" not found Nov 04 00:57:29.440420 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440346 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:30.440328364 +0000 UTC m=+31.629495784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:29.440420 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440347 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:29.440420 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.440377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:29.440420 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.440415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:29.440618 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440459 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:57:29.440618 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440472 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls podName:8cf9dde4-e870-49c8-a34d-1c0a4c85acb6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:30.440458779 +0000 UTC m=+31.629626201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69rhn" (UID: "8cf9dde4-e870-49c8-a34d-1c0a4c85acb6") : secret "samples-operator-tls" not found Nov 04 00:57:29.440618 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440494 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:30.440480053 +0000 UTC m=+31.629647467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : secret "router-metrics-certs-default" not found Nov 04 00:57:29.440618 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440541 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:29.440618 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440558 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c85b76dc-nfpkl: secret "image-registry-tls" not found Nov 04 00:57:29.440618 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:29.440605 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls podName:3603f9b4-53ae-4e45-832f-e5c29ca743b9 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:30.440592273 +0000 UTC m=+31.629759702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls") pod "image-registry-75c85b76dc-nfpkl" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9") : secret "image-registry-tls" not found Nov 04 00:57:29.446628 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.446593 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vl6w6"] Nov 04 00:57:29.448441 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.448418 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5"] Nov 04 00:57:29.451724 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:29.451699 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbd7a56_ab21_413c_abac_36b1f79f452e.slice/crio-aff40f30cc8a15b11e4219db1c765a0ca2efe27e9cfd477651e18db5871fb28d WatchSource:0}: Error finding container aff40f30cc8a15b11e4219db1c765a0ca2efe27e9cfd477651e18db5871fb28d: Status 404 returned error can't find the container with id aff40f30cc8a15b11e4219db1c765a0ca2efe27e9cfd477651e18db5871fb28d Nov 04 00:57:29.452377 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:29.452358 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969ca7d8_004d_4f22_9929_96099b5f011c.slice/crio-f4fa09327650c9e2e4b2ddfaff4f968d1b740be1c68bf83d0805136a22ede284 WatchSource:0}: Error finding container f4fa09327650c9e2e4b2ddfaff4f968d1b740be1c68bf83d0805136a22ede284: Status 404 returned error can't find the container with id f4fa09327650c9e2e4b2ddfaff4f968d1b740be1c68bf83d0805136a22ede284 Nov 04 00:57:29.526583 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.526548 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" event={"ID":"ede327f8-e1cf-4596-abe5-de379f528b15","Type":"ContainerStarted","Data":"5311f6e8245b83b66782847327555239f801dec8e83745900c465a9ddb43afef"} Nov 04 00:57:29.528105 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.528075 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" event={"ID":"767984a4-597e-4498-a6bf-c988a7b228d1","Type":"ContainerStarted","Data":"2ce9bbdc96f0b1eb244987ab23aed35aeee4a5c0249b097882464a6f6f621ce6"} Nov 04 00:57:29.528240 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.528138 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" event={"ID":"767984a4-597e-4498-a6bf-c988a7b228d1","Type":"ContainerStarted","Data":"f688ab99c75d6ab87c4e41c21a5ea21bbe92b9f815cf6d575a7e673601e1f7ae"} Nov 04 00:57:29.531734 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.531691 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" event={"ID":"6aa9b05c-8730-4590-8709-0275e8be2631","Type":"ContainerStarted","Data":"26a695a2da2a162d102bce6a783823e17516262cd423f2298bed353ef5b1bec5"} Nov 04 00:57:29.533166 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.533144 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" event={"ID":"e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd","Type":"ContainerStarted","Data":"57183baeb6c2daf9e4b7d2a79f64a1e6fbbb217c15e01a697909955d5247a5e5"} Nov 04 00:57:29.534555 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.534535 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" event={"ID":"ac81ee2f-5c6a-4ead-89c5-2a181b928494","Type":"ContainerStarted","Data":"401199cd7b1c298d8c5670a3864d7af334e516b4f76416d595ebfa60a3f84735"} Nov 04 00:57:29.534653 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.534573 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" event={"ID":"ac81ee2f-5c6a-4ead-89c5-2a181b928494","Type":"ContainerStarted","Data":"afdc3c7a78e87a2dfa2d67a2961090924b7afe530fe9ecc441f15e941397a578"} Nov 04 00:57:29.536214 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.536186 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" event={"ID":"f4bef706-ca6f-4260-962f-ad0599ad093a","Type":"ContainerStarted","Data":"0543ca44c733cebc767907667eee7a3bedf8f4aa587f854957463e5466c9f483"} Nov 04 00:57:29.536329 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.536216 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" event={"ID":"f4bef706-ca6f-4260-962f-ad0599ad093a","Type":"ContainerStarted","Data":"920363e7ada1def36659576aa31abf5fc2cbcef52f5bd8a5eaf6dc80ba6371d2"} Nov 04 00:57:29.537236 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.537218 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vl6w6" event={"ID":"8dbd7a56-ab21-413c-abac-36b1f79f452e","Type":"ContainerStarted","Data":"aff40f30cc8a15b11e4219db1c765a0ca2efe27e9cfd477651e18db5871fb28d"} Nov 04 00:57:29.538244 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.538226 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" event={"ID":"9747f7d7-39f9-452d-9e9a-dbc9de48c89e","Type":"ContainerStarted","Data":"241c72194a633d194bd11f6c7701f43879c5912719b72bd31b9c38201f356830"} Nov 04 00:57:29.539226 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.539208 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" event={"ID":"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86","Type":"ContainerStarted","Data":"827a6a1b277083251715d55f9a2bcf3d78f6fc9f06ace303c578bd348ad12944"} Nov 04 00:57:29.540275 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.540259 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" event={"ID":"969ca7d8-004d-4f22-9929-96099b5f011c","Type":"ContainerStarted","Data":"f4fa09327650c9e2e4b2ddfaff4f968d1b740be1c68bf83d0805136a22ede284"} Nov 04 00:57:29.549873 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.549835 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" podStartSLOduration=444.549823848 podStartE2EDuration="7m24.549823848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:29.549594152 +0000 UTC m=+30.738761608" watchObservedRunningTime="2026-11-04 00:57:29.549823848 +0000 UTC m=+30.738991285" Nov 04 00:57:29.568582 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.568534 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" podStartSLOduration=444.568521231 podStartE2EDuration="7m24.568521231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:29.567564776 +0000 UTC m=+30.756732214" watchObservedRunningTime="2026-11-04 00:57:29.568521231 +0000 UTC m=+30.757688667" Nov 04 00:57:29.585806 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:29.585758 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" podStartSLOduration=444.585745459 podStartE2EDuration="7m24.585745459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:29.585677837 +0000 UTC m=+30.774845274" watchObservedRunningTime="2026-11-04 00:57:29.585745459 +0000 UTC m=+30.774912934" Nov 04 00:57:30.272100 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.270423 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:30.272100 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.270986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:30.272100 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.271492 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:30.291543 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.291508 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q48lg\"" Nov 04 00:57:30.292059 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.291594 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nszlw\"" Nov 04 00:57:30.292059 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.291639 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:57:30.292059 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.291744 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4rmxc\"" Nov 04 00:57:30.292695 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.292671 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:57:30.350282 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.349381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:30.350282 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.349548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:30.350282 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.349715 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:30.350282 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.349778 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls podName:d648b600-369c-47b0-ba02-6b3df1252bed nodeName:}" failed. No retries permitted until 2026-11-04 00:57:32.349757657 +0000 UTC m=+33.538925077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kxzp6" (UID: "d648b600-369c-47b0-ba02-6b3df1252bed") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:30.350282 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.350203 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:57:30.350282 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.350255 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert podName:261f5fa6-5956-4a5d-a104-982dc13aed39 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:32.350240042 +0000 UTC m=+33.539407458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mh5d6" (UID: "261f5fa6-5956-4a5d-a104-982dc13aed39") : secret "networking-console-plugin-cert" not found Nov 04 00:57:30.450700 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.450663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:30.450860 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.450738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:30.450860 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.450770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:30.450860 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.450856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:30.451033 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.450891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:30.451033 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:30.450921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:30.451195 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.451143 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:30.451249 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.451206 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls podName:8cf9dde4-e870-49c8-a34d-1c0a4c85acb6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:32.451187427 +0000 UTC m=+33.640354850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69rhn" (UID: "8cf9dde4-e870-49c8-a34d-1c0a4c85acb6") : secret "samples-operator-tls" not found Nov 04 00:57:30.451975 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.451954 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:30.451975 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.451976 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bfcf87bc8-zhslb: secret "image-registry-tls" not found Nov 04 00:57:30.451975 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452018 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls podName:e54204af-fcd5-442e-9009-f58ba803f3c4 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:32.452003412 +0000 UTC m=+33.641170834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls") pod "image-registry-6bfcf87bc8-zhslb" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4") : secret "image-registry-tls" not found Nov 04 00:57:30.452320 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452074 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:57:30.452320 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452105 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:32.452094357 +0000 UTC m=+33.641261777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : secret "router-metrics-certs-default" not found Nov 04 00:57:30.452320 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452173 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:30.452320 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452183 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c85b76dc-nfpkl: secret "image-registry-tls" not found Nov 04 00:57:30.452320 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452213 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls podName:3603f9b4-53ae-4e45-832f-e5c29ca743b9 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:32.452201792 +0000 UTC m=+33.641369210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls") pod "image-registry-75c85b76dc-nfpkl" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9") : secret "image-registry-tls" not found Nov 04 00:57:30.452320 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452265 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:30.452320 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452292 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls podName:b0807c2d-b756-42e8-86b3-97c30f23df72 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:32.452282068 +0000 UTC m=+33.641449488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls") pod "dns-default-mvtzk" (UID: "b0807c2d-b756-42e8-86b3-97c30f23df72") : secret "dns-default-metrics-tls" not found Nov 04 00:57:30.452620 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:30.452349 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:32.45233777 +0000 UTC m=+33.641505187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:32.373043 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.372974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:32.373556 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.373139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:32.373556 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.373355 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:32.373556 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.373409 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls podName:d648b600-369c-47b0-ba02-6b3df1252bed nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.373396497 +0000 UTC m=+37.562563912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kxzp6" (UID: "d648b600-369c-47b0-ba02-6b3df1252bed") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:32.373775 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.373758 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:57:32.373830 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.373807 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert podName:261f5fa6-5956-4a5d-a104-982dc13aed39 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.373794471 +0000 UTC m=+37.562961908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mh5d6" (UID: "261f5fa6-5956-4a5d-a104-982dc13aed39") : secret "networking-console-plugin-cert" not found Nov 04 00:57:32.473875 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.473834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:32.473875 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.473881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:32.474105 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.473993 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.473975931 +0000 UTC m=+37.663143363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:32.474105 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.473996 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:32.474105 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.474012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:32.474105 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474058 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls podName:b0807c2d-b756-42e8-86b3-97c30f23df72 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.474040861 +0000 UTC m=+37.663208279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls") pod "dns-default-mvtzk" (UID: "b0807c2d-b756-42e8-86b3-97c30f23df72") : secret "dns-default-metrics-tls" not found Nov 04 00:57:32.474105 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474100 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:32.474345 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474151 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls podName:8cf9dde4-e870-49c8-a34d-1c0a4c85acb6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.47414016 +0000 UTC m=+37.663307591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69rhn" (UID: "8cf9dde4-e870-49c8-a34d-1c0a4c85acb6") : secret "samples-operator-tls" not found Nov 04 00:57:32.474345 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.474208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:32.474345 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.474261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:32.474345 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.474286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:32.474537 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474357 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:32.474537 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474372 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bfcf87bc8-zhslb: secret "image-registry-tls" not found Nov 04 00:57:32.474537 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474377 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:32.474537 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474389 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c85b76dc-nfpkl: secret "image-registry-tls" not found Nov 04 00:57:32.474537 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474411 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls podName:e54204af-fcd5-442e-9009-f58ba803f3c4 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.47439906 +0000 UTC m=+37.663566493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls") pod "image-registry-6bfcf87bc8-zhslb" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4") : secret "image-registry-tls" not found Nov 04 00:57:32.474537 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474428 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls podName:3603f9b4-53ae-4e45-832f-e5c29ca743b9 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.474419262 +0000 UTC m=+37.663586683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls") pod "image-registry-75c85b76dc-nfpkl" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9") : secret "image-registry-tls" not found Nov 04 00:57:32.474537 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474443 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:57:32.474537 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:32.474486 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.474472575 +0000 UTC m=+37.663640006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : secret "router-metrics-certs-default" not found Nov 04 00:57:32.555632 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.555594 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" event={"ID":"e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd","Type":"ContainerStarted","Data":"1229ba3f8dac1a2d5e5b70e3755cfb5e0a444d08aaa97aba4075cb446fcb85e7"} Nov 04 00:57:32.572362 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:32.572308 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" podStartSLOduration=384.329632056 podStartE2EDuration="6m26.57227986s" totalImagesPullingTime="2.242647804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:32.571041758 +0000 UTC m=+33.760209196" watchObservedRunningTime="2026-11-04 00:57:32.57227986 +0000 UTC m=+33.761447297" Nov 04 00:57:33.081138 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.081074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:33.081313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.081239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:33.081313 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.081286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:33.081840 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:33.081816 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 00:57:33.081933 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:33.081887 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:58:05.081870622 +0000 UTC m=+66.271038042 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : secret "metrics-daemon-secret" not found Nov 04 00:57:33.087864 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.087788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:33.089871 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.089848 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:33.343877 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.343797 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:33.358991 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.358960 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:33.562947 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.562909 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" event={"ID":"9747f7d7-39f9-452d-9e9a-dbc9de48c89e","Type":"ContainerStarted","Data":"7ad7c42100faf530ca8cac78fca552cc91afc47e31f9b7b6d2593f1da3514ecb"} Nov 04 00:57:33.656876 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.656847 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8rwkl"] Nov 04 00:57:33.659221 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:33.659191 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f82b22_3949_44d3_abe9_6473a2f68f76.slice/crio-4dbb831cc7b9913eb4c21b4a7e7a8d7246d65af7722c61405ed3e6363c7330b4 WatchSource:0}: Error finding container 4dbb831cc7b9913eb4c21b4a7e7a8d7246d65af7722c61405ed3e6363c7330b4: Status 404 returned error can't find the container with id 4dbb831cc7b9913eb4c21b4a7e7a8d7246d65af7722c61405ed3e6363c7330b4 Nov 04 00:57:33.688530 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:33.688502 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zh5bh"] Nov 04 00:57:33.690664 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:33.690627 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c16477_f3db_43cf_9830_39fe0267bdce.slice/crio-3df5bb11ab6ab6c7a2827925a728e7a8668d6af074c86473c65e094509680e32 WatchSource:0}: Error finding container 3df5bb11ab6ab6c7a2827925a728e7a8668d6af074c86473c65e094509680e32: Status 404 returned error can't find the container with id 3df5bb11ab6ab6c7a2827925a728e7a8668d6af074c86473c65e094509680e32 Nov 04 00:57:34.568685 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.568656 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/0.log" Nov 04 00:57:34.569147 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.568699 2578 generic.go:356] "Generic (PLEG): container finished" podID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" containerID="7ad7c42100faf530ca8cac78fca552cc91afc47e31f9b7b6d2593f1da3514ecb" exitCode=255 Nov 04 00:57:34.569147 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.568772 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" event={"ID":"9747f7d7-39f9-452d-9e9a-dbc9de48c89e","Type":"ContainerDied","Data":"7ad7c42100faf530ca8cac78fca552cc91afc47e31f9b7b6d2593f1da3514ecb"} Nov 04 00:57:34.569147 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.569039 2578 scope.go:122] "RemoveContainer" containerID="7ad7c42100faf530ca8cac78fca552cc91afc47e31f9b7b6d2593f1da3514ecb" Nov 04 00:57:34.570934 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.570911 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" event={"ID":"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86","Type":"ContainerStarted","Data":"722e517c22708269134694f3f5d833ed958ae5d513927fd9c6d3bdab9056f6a3"} Nov 04 00:57:34.572766 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.572744 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8rwkl" event={"ID":"04f82b22-3949-44d3-abe9-6473a2f68f76","Type":"ContainerStarted","Data":"4dbb831cc7b9913eb4c21b4a7e7a8d7246d65af7722c61405ed3e6363c7330b4"} Nov 04 00:57:34.575217 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.575165 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" event={"ID":"ede327f8-e1cf-4596-abe5-de379f528b15","Type":"ContainerStarted","Data":"00e74c9e1ed136bcdb85b5aef14228d27ea9e91ab2058c6cf4dc6df1668155c1"} Nov 04 00:57:34.576717 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.576695 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zh5bh" event={"ID":"66c16477-f3db-43cf-9830-39fe0267bdce","Type":"ContainerStarted","Data":"3df5bb11ab6ab6c7a2827925a728e7a8668d6af074c86473c65e094509680e32"} Nov 04 00:57:34.605532 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.605483 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" podStartSLOduration=379.728107232 podStartE2EDuration="6m24.605470453s" totalImagesPullingTime="4.877363221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:34.605035436 +0000 UTC m=+35.794202874" watchObservedRunningTime="2026-11-04 00:57:34.605470453 +0000 UTC m=+35.794637890" Nov 04 00:57:34.626072 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:34.626012 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" podStartSLOduration=384.784990572 podStartE2EDuration="6m29.625994389s" totalImagesPullingTime="4.841003817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:34.625088788 +0000 UTC m=+35.814256226" watchObservedRunningTime="2026-11-04 00:57:34.625994389 +0000 UTC m=+35.815161831" Nov 04 00:57:35.058921 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.058887 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m"] Nov 04 00:57:35.082642 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.082248 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh"] Nov 04 00:57:35.083149 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.082378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" Nov 04 00:57:35.093448 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.090974 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:35.093448 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.092404 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-djh2q\"" Nov 04 00:57:35.093448 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.091114 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 00:57:35.103611 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.103558 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m"] Nov 04 00:57:35.103870 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.103642 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh"] Nov 04 00:57:35.103870 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.103792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" Nov 04 00:57:35.169642 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.169610 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wtpbl"] Nov 04 00:57:35.187428 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.187368 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tw9hp-q2fgs"] Nov 04 00:57:35.188816 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.188795 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.193319 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.193297 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:57:35.193984 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.193934 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldjm6\"" Nov 04 00:57:35.199514 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.199471 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:57:35.204466 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.204218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8d5k\" (UniqueName: \"kubernetes.io/projected/756d0c06-26e9-4c88-8520-34f21fa6a9bd-kube-api-access-w8d5k\") pod \"migrator-66df5d496b-n84lh\" (UID: \"756d0c06-26e9-4c88-8520-34f21fa6a9bd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" Nov 04 00:57:35.204466 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.204277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swhr9\" (UniqueName: \"kubernetes.io/projected/903f3694-f0a7-4d5f-828b-20bced1b1066-kube-api-access-swhr9\") pod \"migrator-66df5d496b-lbr5m\" (UID: \"903f3694-f0a7-4d5f-828b-20bced1b1066\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" Nov 04 00:57:35.210210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.209875 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wtpbl"] Nov 04 00:57:35.210210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.209926 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tw9hp-q2fgs"] Nov 04 00:57:35.210210 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.210033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.305437 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhr9\" (UniqueName: \"kubernetes.io/projected/903f3694-f0a7-4d5f-828b-20bced1b1066-kube-api-access-swhr9\") pod \"migrator-66df5d496b-lbr5m\" (UID: \"903f3694-f0a7-4d5f-828b-20bced1b1066\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" Nov 04 00:57:35.305553 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305456 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.305553 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.305553 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.305701 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305585 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-data-volume\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.305701 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk25v\" (UniqueName: \"kubernetes.io/projected/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-api-access-hk25v\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.305790 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.305790 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-crio-socket\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.305890 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnff8\" (UniqueName: \"kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.305967 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.306017 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.305988 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8d5k\" (UniqueName: \"kubernetes.io/projected/756d0c06-26e9-4c88-8520-34f21fa6a9bd-kube-api-access-w8d5k\") pod \"migrator-66df5d496b-n84lh\" (UID: \"756d0c06-26e9-4c88-8520-34f21fa6a9bd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" Nov 04 00:57:35.314907 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.314884 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhr9\" (UniqueName: \"kubernetes.io/projected/903f3694-f0a7-4d5f-828b-20bced1b1066-kube-api-access-swhr9\") pod \"migrator-66df5d496b-lbr5m\" (UID: \"903f3694-f0a7-4d5f-828b-20bced1b1066\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" Nov 04 00:57:35.315889 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.315869 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8d5k\" (UniqueName: \"kubernetes.io/projected/756d0c06-26e9-4c88-8520-34f21fa6a9bd-kube-api-access-w8d5k\") pod \"migrator-66df5d496b-n84lh\" (UID: \"756d0c06-26e9-4c88-8520-34f21fa6a9bd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.406809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.406844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.406892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-data-volume\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.406924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk25v\" (UniqueName: \"kubernetes.io/projected/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-api-access-hk25v\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.406950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.406967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-crio-socket\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.406982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnff8\" (UniqueName: \"kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.407016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.407162 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.407054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.407723 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.407190 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:35.407723 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.407265 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:35.907242295 +0000 UTC m=+37.096409728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:35.407723 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.407618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.407723 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.407710 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:35.407913 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.407758 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:35.907742947 +0000 UTC m=+37.096910382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : configmap "kube-rbac-proxy" not found Nov 04 00:57:35.408588 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.408036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-data-volume\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.408588 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.408062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-crio-socket\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.408588 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.408540 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.408942 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.408920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.418234 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.418184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnff8\" (UniqueName: \"kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.422460 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.422416 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk25v\" (UniqueName: \"kubernetes.io/projected/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-api-access-hk25v\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.431595 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.431429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" Nov 04 00:57:35.451029 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.451005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" Nov 04 00:57:35.582452 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.581884 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:35.585171 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.584510 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vl6w6" event={"ID":"8dbd7a56-ab21-413c-abac-36b1f79f452e","Type":"ContainerStarted","Data":"e0b02d505074afb48cc3f5d141dfa1ade1341f13c2d5b6cc9828e93983853b82"} Nov 04 00:57:35.592497 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.592470 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 00:57:35.593216 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.593191 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/0.log" Nov 04 00:57:35.593340 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.593243 2578 generic.go:356] "Generic (PLEG): container finished" podID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" containerID="8b1ac2c33f7a81e7550c8cc5e675d9803e8f2307b5e6c85fbbd6e1a0817c6094" exitCode=255 Nov 04 00:57:35.593456 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.593331 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" event={"ID":"9747f7d7-39f9-452d-9e9a-dbc9de48c89e","Type":"ContainerDied","Data":"8b1ac2c33f7a81e7550c8cc5e675d9803e8f2307b5e6c85fbbd6e1a0817c6094"} Nov 04 00:57:35.593456 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.593390 2578 scope.go:122] "RemoveContainer" containerID="7ad7c42100faf530ca8cac78fca552cc91afc47e31f9b7b6d2593f1da3514ecb" Nov 04 00:57:35.594070 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.593612 2578 scope.go:122] "RemoveContainer" containerID="8b1ac2c33f7a81e7550c8cc5e675d9803e8f2307b5e6c85fbbd6e1a0817c6094" Nov 04 00:57:35.594070 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.593850 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vtl7_openshift-console-operator(9747f7d7-39f9-452d-9e9a-dbc9de48c89e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" Nov 04 00:57:35.596535 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.596510 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" event={"ID":"969ca7d8-004d-4f22-9929-96099b5f011c","Type":"ContainerStarted","Data":"5caf8bd28178c7062a93e707fbdb952c900fe519f7350cf707f40cf29a2eddf7"} Nov 04 00:57:35.599527 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.599501 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" event={"ID":"6aa9b05c-8730-4590-8709-0275e8be2631","Type":"ContainerStarted","Data":"c328e5385563653eddd50c01ddd929a6663093428d488cf01b874d042c1ab541"} Nov 04 00:57:35.619826 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.619770 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" podStartSLOduration=384.194860828 podStartE2EDuration="6m29.619751219s" totalImagesPullingTime="5.424890391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:35.618483422 +0000 UTC m=+36.807650862" watchObservedRunningTime="2026-11-04 00:57:35.619751219 +0000 UTC m=+36.808918658" Nov 04 00:57:35.620322 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.620282 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vl6w6" podStartSLOduration=2.21514647 podStartE2EDuration="7.620272358s" totalImagesPullingTime="5.405125888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:35.602275941 +0000 UTC m=+36.791443379" watchObservedRunningTime="2026-11-04 00:57:35.620272358 +0000 UTC m=+36.809439796" Nov 04 00:57:35.661146 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.661086 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" podStartSLOduration=379.87806466 podStartE2EDuration="6m25.66107241s" totalImagesPullingTime="5.78300775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:35.658987549 +0000 UTC m=+36.848154986" watchObservedRunningTime="2026-11-04 00:57:35.66107241 +0000 UTC m=+36.850239847" Nov 04 00:57:35.761155 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.758859 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m"] Nov 04 00:57:35.773765 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:35.773736 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903f3694_f0a7_4d5f_828b_20bced1b1066.slice/crio-f8a3af112af34fc6dbb6d93c1b656c2066586e75788a6f61a43fe5acdfbf8062 WatchSource:0}: Error finding container f8a3af112af34fc6dbb6d93c1b656c2066586e75788a6f61a43fe5acdfbf8062: Status 404 returned error can't find the container with id f8a3af112af34fc6dbb6d93c1b656c2066586e75788a6f61a43fe5acdfbf8062 Nov 04 00:57:35.791274 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.791249 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh"] Nov 04 00:57:35.795093 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:35.795070 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756d0c06_26e9_4c88_8520_34f21fa6a9bd.slice/crio-50c26ad6cc1cb7efb91a498af2c08443e1e80c83ddcca6f0945f31f6fb955e50 WatchSource:0}: Error finding container 50c26ad6cc1cb7efb91a498af2c08443e1e80c83ddcca6f0945f31f6fb955e50: Status 404 returned error can't find the container with id 50c26ad6cc1cb7efb91a498af2c08443e1e80c83ddcca6f0945f31f6fb955e50 Nov 04 00:57:35.911215 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.908093 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tw9hp-q2fgs"] Nov 04 00:57:35.911215 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:35.911167 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9027aeb_9f01_4ef8_b936_15c9382b2af8.slice/crio-0a940fb409c7170d592362e00ec0c2beadf3a50c2102d26c10bd1c05dd6dad67 WatchSource:0}: Error finding container 0a940fb409c7170d592362e00ec0c2beadf3a50c2102d26c10bd1c05dd6dad67: Status 404 returned error can't find the container with id 0a940fb409c7170d592362e00ec0c2beadf3a50c2102d26c10bd1c05dd6dad67 Nov 04 00:57:35.913685 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.913661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.913802 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:35.913713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:35.913872 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.913801 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:35.913872 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.913826 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:35.913872 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.913865 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.913847365 +0000 UTC m=+38.103014785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : configmap "kube-rbac-proxy" not found Nov 04 00:57:35.914007 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:35.913884 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:36.913874157 +0000 UTC m=+38.103041572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:36.299620 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.299589 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2g6px"] Nov 04 00:57:36.320818 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.320791 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2g6px"] Nov 04 00:57:36.320980 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.320901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.324908 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.324868 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-52wlr\"" Nov 04 00:57:36.324908 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.324882 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 00:57:36.324908 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.324869 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 00:57:36.325155 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.324868 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 00:57:36.325155 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.324912 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 00:57:36.420196 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.420152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:36.420364 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.420274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6b48\" (UniqueName: \"kubernetes.io/projected/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-kube-api-access-k6b48\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.420412 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.420364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:36.420412 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.420392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-key\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.420501 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.420426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.421198 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.420596 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:36.421198 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.420658 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls podName:d648b600-369c-47b0-ba02-6b3df1252bed nodeName:}" failed. No retries permitted until 2026-11-04 00:57:44.420639861 +0000 UTC m=+45.609807282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-kxzp6" (UID: "d648b600-369c-47b0-ba02-6b3df1252bed") : secret "cluster-monitoring-operator-tls" not found Nov 04 00:57:36.421198 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.421063 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:57:36.421198 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.421108 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert podName:261f5fa6-5956-4a5d-a104-982dc13aed39 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:44.421093809 +0000 UTC m=+45.610261230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mh5d6" (UID: "261f5fa6-5956-4a5d-a104-982dc13aed39") : secret "networking-console-plugin-cert" not found Nov 04 00:57:36.520962 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.520927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:36.521102 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.521018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:36.521102 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.521047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:36.521102 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.521078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6b48\" (UniqueName: \"kubernetes.io/projected/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-kube-api-access-k6b48\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.521291 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.521105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:36.521291 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.521155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:36.521291 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.521180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:36.521291 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.521222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-key\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.521291 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.521245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.521808 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.521847 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:44.521824684 +0000 UTC m=+45.710992116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.521912 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls podName:8cf9dde4-e870-49c8-a34d-1c0a4c85acb6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:44.52189528 +0000 UTC m=+45.711062715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69rhn" (UID: "8cf9dde4-e870-49c8-a34d-1c0a4c85acb6") : secret "samples-operator-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522029 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522042 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522072 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:44.522058392 +0000 UTC m=+45.711225825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : secret "router-metrics-certs-default" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522087 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls podName:b0807c2d-b756-42e8-86b3-97c30f23df72 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:44.522079529 +0000 UTC m=+45.711246949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls") pod "dns-default-mvtzk" (UID: "b0807c2d-b756-42e8-86b3-97c30f23df72") : secret "dns-default-metrics-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522099 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522110 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bfcf87bc8-zhslb: secret "image-registry-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522158 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522161 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls podName:e54204af-fcd5-442e-9009-f58ba803f3c4 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:44.522150632 +0000 UTC m=+45.711318061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls") pod "image-registry-6bfcf87bc8-zhslb" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4") : secret "image-registry-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522168 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c85b76dc-nfpkl: secret "image-registry-tls" not found Nov 04 00:57:36.522293 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.522198 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls podName:3603f9b4-53ae-4e45-832f-e5c29ca743b9 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:44.522187817 +0000 UTC m=+45.711355235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls") pod "image-registry-75c85b76dc-nfpkl" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9") : secret "image-registry-tls" not found Nov 04 00:57:36.523281 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.522627 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.524868 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.524838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-key\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.534144 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.534104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6b48\" (UniqueName: \"kubernetes.io/projected/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-kube-api-access-k6b48\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.608926 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.608798 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" event={"ID":"756d0c06-26e9-4c88-8520-34f21fa6a9bd","Type":"ContainerStarted","Data":"50c26ad6cc1cb7efb91a498af2c08443e1e80c83ddcca6f0945f31f6fb955e50"} Nov 04 00:57:36.611135 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.611095 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" event={"ID":"903f3694-f0a7-4d5f-828b-20bced1b1066","Type":"ContainerStarted","Data":"f8a3af112af34fc6dbb6d93c1b656c2066586e75788a6f61a43fe5acdfbf8062"} Nov 04 00:57:36.613684 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.613661 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 00:57:36.614848 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.614115 2578 scope.go:122] "RemoveContainer" containerID="8b1ac2c33f7a81e7550c8cc5e675d9803e8f2307b5e6c85fbbd6e1a0817c6094" Nov 04 00:57:36.614848 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.614367 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vtl7_openshift-console-operator(9747f7d7-39f9-452d-9e9a-dbc9de48c89e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" Nov 04 00:57:36.619651 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.618000 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" event={"ID":"c9027aeb-9f01-4ef8-b936-15c9382b2af8","Type":"ContainerStarted","Data":"2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab"} Nov 04 00:57:36.619651 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.618042 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" event={"ID":"c9027aeb-9f01-4ef8-b936-15c9382b2af8","Type":"ContainerStarted","Data":"0a940fb409c7170d592362e00ec0c2beadf3a50c2102d26c10bd1c05dd6dad67"} Nov 04 00:57:36.645348 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.645321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:36.658894 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.658838 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" podStartSLOduration=1.6588193260000001 podStartE2EDuration="1.658819326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:36.658225341 +0000 UTC m=+37.847392781" watchObservedRunningTime="2026-11-04 00:57:36.658819326 +0000 UTC m=+37.847986763" Nov 04 00:57:36.932898 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.932804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:36.932898 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.932890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:36.933107 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.933051 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:36.933189 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.933147 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:38.933104977 +0000 UTC m=+40.122272408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : configmap "kube-rbac-proxy" not found Nov 04 00:57:36.933640 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.933593 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:36.933703 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:36.933642 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:38.933628773 +0000 UTC m=+40.122796218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:36.977631 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:36.977597 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2g6px"] Nov 04 00:57:36.981691 ip-10-0-137-48 kubenswrapper[2578]: W1104 00:57:36.981653 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3b4dfd_5d4c_4ac6_b46f_17a9755f03ac.slice/crio-d5158317b1a64306381ef738db0f9d6b565351eb291be2943ecd9390e6b2b1f8 WatchSource:0}: Error finding container d5158317b1a64306381ef738db0f9d6b565351eb291be2943ecd9390e6b2b1f8: Status 404 returned error can't find the container with id d5158317b1a64306381ef738db0f9d6b565351eb291be2943ecd9390e6b2b1f8 Nov 04 00:57:37.625412 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:37.625304 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" event={"ID":"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac","Type":"ContainerStarted","Data":"a466679c253cf202577a395f93edfca924afa4b09a575c2477321f502e1b57b7"} Nov 04 00:57:37.625949 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:37.625425 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" event={"ID":"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac","Type":"ContainerStarted","Data":"d5158317b1a64306381ef738db0f9d6b565351eb291be2943ecd9390e6b2b1f8"} Nov 04 00:57:37.649061 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:37.649004 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" podStartSLOduration=1.6489602890000001 podStartE2EDuration="1.648960289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:37.646428732 +0000 UTC m=+38.835596169" watchObservedRunningTime="2026-11-04 00:57:37.648960289 +0000 UTC m=+38.838127727" Nov 04 00:57:38.214150 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.214060 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-shjsf_51ff9909-e56c-4d27-be31-574bf33a8f75/dns-node-resolver/0.log" Nov 04 00:57:38.629337 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.629256 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zh5bh" event={"ID":"66c16477-f3db-43cf-9830-39fe0267bdce","Type":"ContainerStarted","Data":"9694587feb9db170336d7ae7838ba03d9d0692d25644bc19330723abaaf8cf2d"} Nov 04 00:57:38.629765 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.629453 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:38.630999 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.630975 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" event={"ID":"903f3694-f0a7-4d5f-828b-20bced1b1066","Type":"ContainerStarted","Data":"35163177bec0ecc297b9da3a7e2a43b9cf33223c59182b887d4082495672b187"} Nov 04 00:57:38.632596 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.632572 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" event={"ID":"756d0c06-26e9-4c88-8520-34f21fa6a9bd","Type":"ContainerStarted","Data":"c401df772ccf259a9d7dc2a667017dc6cb1c6af9bebf249c73f868096cba959f"} Nov 04 00:57:38.649305 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.649000 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zh5bh" podStartSLOduration=35.093112122 podStartE2EDuration="39.64898372s" totalImagesPullingTime="4.555871598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:38.648002796 +0000 UTC m=+39.837170232" watchObservedRunningTime="2026-11-04 00:57:38.64898372 +0000 UTC m=+39.838151135" Nov 04 00:57:38.676577 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.676515 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" podStartSLOduration=1.476050703 podStartE2EDuration="3.67649652s" totalImagesPullingTime="2.200445817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:38.67455841 +0000 UTC m=+39.863725860" watchObservedRunningTime="2026-11-04 00:57:38.67649652 +0000 UTC m=+39.865663957" Nov 04 00:57:38.727100 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.727038 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" podStartSLOduration=1.510669995 podStartE2EDuration="3.727020704s" totalImagesPullingTime="2.216350709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:38.726805066 +0000 UTC m=+39.915972514" watchObservedRunningTime="2026-11-04 00:57:38.727020704 +0000 UTC m=+39.916188138" Nov 04 00:57:38.808045 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.808009 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcksb_5b7cc6df-efb4-4ed5-a299-99d3eafd8e56/node-ca/0.log" Nov 04 00:57:38.959855 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.959766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:38.960026 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:38.959863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:38.960026 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:38.959938 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:38.960026 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:38.959978 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 00:57:38.960026 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:38.960011 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.959990562 +0000 UTC m=+44.149157999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:38.960250 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:38.960033 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.960023788 +0000 UTC m=+44.149191234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : configmap "kube-rbac-proxy" not found Nov 04 00:57:39.083887 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:39.083843 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:39.084076 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:39.083945 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:39.084460 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:39.084435 2578 scope.go:122] "RemoveContainer" containerID="8b1ac2c33f7a81e7550c8cc5e675d9803e8f2307b5e6c85fbbd6e1a0817c6094" Nov 04 00:57:39.084705 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:39.084683 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vtl7_openshift-console-operator(9747f7d7-39f9-452d-9e9a-dbc9de48c89e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" Nov 04 00:57:39.406461 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:39.406426 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vl6w6_8dbd7a56-ab21-413c-abac-36b1f79f452e/serve-healthcheck-canary/0.log" Nov 04 00:57:39.606937 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:39.606907 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lbr5m_903f3694-f0a7-4d5f-828b-20bced1b1066/migrator/0.log" Nov 04 00:57:39.636641 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:39.636613 2578 scope.go:122] "RemoveContainer" containerID="8b1ac2c33f7a81e7550c8cc5e675d9803e8f2307b5e6c85fbbd6e1a0817c6094" Nov 04 00:57:39.637030 ip-10-0-137-48 kubenswrapper[2578]: E1104 00:57:39.636775 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-7vtl7_openshift-console-operator(9747f7d7-39f9-452d-9e9a-dbc9de48c89e)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" Nov 04 00:57:39.807751 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:39.807719 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-n84lh_756d0c06-26e9-4c88-8520-34f21fa6a9bd/migrator/0.log" Nov 04 00:57:40.008823 ip-10-0-137-48 kubenswrapper[2578]: I1104 00:57:40.008787 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4_2f2b307f-e5f0-4116-b2cd-2bce47ae5b86/kube-storage-version-migrator-operator/0.log" Nov 04 00:57:40.973392 ip-10-0-137-48 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 00:57:40.989541 ip-10-0-137-48 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 00:57:40.989711 ip-10-0-137-48 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 00:57:40.989764 ip-10-0-137-48 systemd[1]: kubelet.service: Consumed 3.201s CPU time. Nov 04 00:57:40.996922 ip-10-0-137-48 systemd[1]: Starting Kubernetes Kubelet... Nov 04 00:57:41.089734 ip-10-0-137-48 kubenswrapper[5051]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:41.089734 ip-10-0-137-48 kubenswrapper[5051]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 00:57:41.089734 ip-10-0-137-48 kubenswrapper[5051]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:41.089734 ip-10-0-137-48 kubenswrapper[5051]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 00:57:41.091837 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091745 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:41.091837 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091835 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:41.091837 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091839 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:41.091837 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091843 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091846 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091849 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091852 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091854 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091857 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091861 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091863 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091866 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091869 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091872 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091874 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091877 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091880 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091882 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091885 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091887 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091890 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091901 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:41.091975 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091905 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091908 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091910 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091913 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091916 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091919 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091922 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091924 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091927 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091930 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091932 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091935 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091937 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091940 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091942 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091945 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091948 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091950 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091953 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091956 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:41.092499 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091959 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091961 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091964 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091966 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091969 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091973 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091978 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091981 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091984 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091987 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091990 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091993 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091995 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.091998 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092001 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092005 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092008 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092011 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092014 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:41.093104 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092016 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092019 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092021 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092024 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092026 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092033 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092037 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092040 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092044 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092049 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092055 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092062 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092067 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092071 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092075 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092077 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092080 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092083 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092086 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:41.093938 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092088 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092091 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092093 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092097 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092100 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092102 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092105 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092107 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092112 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092115 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092136 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092139 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092142 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092144 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092147 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092150 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092152 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092155 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092158 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092160 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:41.094799 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092163 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092166 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092168 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092171 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092173 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092176 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092924 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092931 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092934 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092937 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092940 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092942 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092945 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092948 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092950 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092956 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092960 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092962 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092965 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092968 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:41.095643 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092972 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092975 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092978 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092980 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092983 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092986 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092988 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092991 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092994 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092996 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.092999 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093003 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093007 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093010 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093014 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093018 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093022 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093026 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093037 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:41.096495 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093042 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093189 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093194 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093197 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093200 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093203 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093206 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093209 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093211 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093214 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093217 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093219 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093222 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093225 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093227 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093230 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093233 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093235 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093238 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:41.097341 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093241 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093244 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093247 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093249 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093252 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093254 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093257 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093260 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093263 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093265 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093268 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093270 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093273 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093284 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093288 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093290 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093295 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093299 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093303 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093306 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:41.098134 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093309 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093312 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093315 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093317 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093320 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093323 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093325 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093329 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093332 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093335 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093338 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093341 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093343 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093346 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093349 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093351 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093355 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093359 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093364 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:41.098992 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093367 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093369 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093372 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093375 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093377 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093380 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093382 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093386 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093389 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093391 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093394 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093396 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093399 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093401 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093404 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093925 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093931 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093935 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093943 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093946 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:41.099804 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093949 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093952 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093955 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093958 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093960 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093963 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093966 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093968 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093971 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093974 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093977 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093980 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093983 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093985 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093988 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093991 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093993 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093996 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.093998 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094001 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:41.100627 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094004 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094007 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094010 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094013 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094015 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094018 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094020 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094023 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094025 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094028 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094032 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094034 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094037 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094040 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094042 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094045 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094047 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094050 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094053 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094055 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:41.101452 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094057 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094060 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094064 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094068 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094071 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094073 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094076 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094079 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094082 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094084 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094087 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094089 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094093 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094096 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094112 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094114 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094132 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094136 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094142 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:41.102289 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094145 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094148 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094150 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094154 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094157 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094160 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094162 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094165 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094168 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094170 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094173 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094175 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094178 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094181 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094183 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094186 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094189 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094191 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094194 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094196 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:41.103062 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094199 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094203 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094206 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094209 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094213 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094216 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094219 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094222 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094225 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094228 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094231 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094234 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094236 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094240 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094242 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094246 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094249 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094252 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094254 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:41.103892 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094257 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094259 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.094262 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094366 5051 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094373 5051 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094379 5051 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094384 5051 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094389 5051 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094393 5051 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094398 5051 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094402 5051 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094405 5051 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094409 5051 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094412 5051 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094416 5051 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094419 5051 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094422 5051 flags.go:64] FLAG: --cgroup-root="" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094425 5051 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094428 5051 flags.go:64] FLAG: --client-ca-file="" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094431 5051 flags.go:64] FLAG: --cloud-provider="external" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094435 5051 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094439 5051 flags.go:64] FLAG: --cluster-domain="" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094442 5051 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094446 5051 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 00:57:41.104503 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094449 5051 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094452 5051 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094456 5051 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094459 5051 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094464 5051 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094467 5051 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094471 5051 flags.go:64] FLAG: --contention-profiling="false" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094474 5051 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094477 5051 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094480 5051 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094483 5051 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094488 5051 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094491 5051 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094494 5051 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094497 5051 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094500 5051 flags.go:64] FLAG: --enable-server="true" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094503 5051 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094507 5051 flags.go:64] FLAG: --event-burst="100" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094510 5051 flags.go:64] FLAG: --event-qps="50" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094513 5051 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094517 5051 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094520 5051 flags.go:64] FLAG: --eviction-hard="" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094524 5051 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094527 5051 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094530 5051 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 00:57:41.105107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094533 5051 flags.go:64] FLAG: --eviction-soft="" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094536 5051 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094539 5051 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094543 5051 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094546 5051 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094549 5051 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094552 5051 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094555 5051 flags.go:64] FLAG: --feature-gates="" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094560 5051 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094564 5051 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094567 5051 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094571 5051 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094575 5051 flags.go:64] FLAG: --healthz-port="10248" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094578 5051 flags.go:64] FLAG: --help="false" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094583 5051 flags.go:64] FLAG: --hostname-override="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094586 5051 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094590 5051 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094593 5051 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094596 5051 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094600 5051 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094603 5051 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094606 5051 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094609 5051 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 00:57:41.105851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094612 5051 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094615 5051 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094618 5051 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094621 5051 flags.go:64] FLAG: --kube-reserved="" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094624 5051 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094627 5051 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094630 5051 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094633 5051 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094636 5051 flags.go:64] FLAG: --lock-file="" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094639 5051 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094642 5051 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094645 5051 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094653 5051 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094656 5051 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094660 5051 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094662 5051 flags.go:64] FLAG: --logging-format="text" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094666 5051 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094669 5051 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094672 5051 flags.go:64] FLAG: --manifest-url="" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094675 5051 flags.go:64] FLAG: --manifest-url-header="" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094681 5051 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094684 5051 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094689 5051 flags.go:64] FLAG: --max-pods="110" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094694 5051 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094698 5051 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 00:57:41.106732 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094701 5051 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094704 5051 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094707 5051 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094710 5051 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094714 5051 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094721 5051 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094724 5051 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094727 5051 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094731 5051 flags.go:64] FLAG: --pod-cidr="" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094734 5051 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094737 5051 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094740 5051 flags.go:64] FLAG: --pods-per-core="0" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094743 5051 flags.go:64] FLAG: --port="10250" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094746 5051 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094749 5051 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0235cc2b8b49474a0" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094752 5051 flags.go:64] FLAG: --qos-reserved="" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094755 5051 flags.go:64] FLAG: --read-only-port="10255" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094758 5051 flags.go:64] FLAG: --register-node="true" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094761 5051 flags.go:64] FLAG: --register-with-taints="" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094766 5051 flags.go:64] FLAG: --registry-burst="10" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094769 5051 flags.go:64] FLAG: --registry-qps="5" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094772 5051 flags.go:64] FLAG: --reserved-cpus="" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094776 5051 flags.go:64] FLAG: --reserved-memory="" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094780 5051 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094783 5051 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094786 5051 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 00:57:41.107502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094789 5051 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094794 5051 flags.go:64] FLAG: --runonce="false" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094797 5051 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094801 5051 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094806 5051 flags.go:64] FLAG: --seccomp-default="false" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094809 5051 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094828 5051 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094833 5051 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094837 5051 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094840 5051 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094843 5051 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094846 5051 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094849 5051 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094852 5051 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094856 5051 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094859 5051 flags.go:64] FLAG: --system-cgroups="" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094862 5051 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094867 5051 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094870 5051 flags.go:64] FLAG: --tls-cert-file="" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094873 5051 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094877 5051 flags.go:64] FLAG: --tls-min-version="" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094880 5051 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094882 5051 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094885 5051 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094889 5051 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 00:57:41.108273 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094892 5051 flags.go:64] FLAG: --v="2" Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094897 5051 flags.go:64] FLAG: --version="false" Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094901 5051 flags.go:64] FLAG: --vmodule="" Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094905 5051 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.094908 5051 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095050 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095054 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095057 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095061 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095064 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095067 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095071 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095074 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095076 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095079 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095081 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095084 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095086 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095089 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095092 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095095 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:41.109075 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095097 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095100 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095103 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095106 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095109 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095112 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095114 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095132 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095136 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095138 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095141 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095143 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095146 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095149 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095151 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095154 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095157 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095161 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095166 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:41.109803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095171 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095174 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095176 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095180 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095183 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095185 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095189 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095193 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095196 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095198 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095201 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095203 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095206 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095208 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095211 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095214 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095216 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095219 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095221 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:41.110306 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095224 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095226 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095229 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095232 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095235 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095237 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095240 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095242 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095245 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095248 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095250 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095253 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095256 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095260 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095262 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095268 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095272 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095275 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095278 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:41.110785 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095280 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095283 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095285 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095288 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095290 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095293 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095296 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095298 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095301 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095304 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095306 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095309 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095311 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095314 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095316 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095319 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095322 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095324 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095327 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095329 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:41.111273 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095332 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095334 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095337 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095340 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095342 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095345 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095349 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095352 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095355 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095358 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095361 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095364 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095366 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.095392 5051 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095428 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095433 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:41.111771 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095436 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095439 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095442 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095444 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095447 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095449 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095452 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095455 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095457 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095460 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095463 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095465 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095468 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095470 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095473 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095475 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095478 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095481 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095483 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:41.112185 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095486 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095488 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095491 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095497 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095501 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095504 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095508 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095510 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095513 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095516 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095518 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095521 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095524 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095526 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095529 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095531 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095534 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095536 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095539 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:41.112687 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095541 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095544 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095547 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095549 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095552 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095554 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095557 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095560 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095562 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095564 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095567 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095570 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095572 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095575 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095577 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095580 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095583 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095588 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095591 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095595 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:41.113225 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095597 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095600 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095602 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095605 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095607 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095610 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095613 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095615 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095618 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095620 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095623 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095626 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095628 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095630 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095633 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095635 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095638 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095641 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095643 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095647 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:41.113728 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095650 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095653 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095656 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095658 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095661 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095664 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095666 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095669 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095671 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095676 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095679 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095683 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095685 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095688 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095692 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095702 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095706 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095709 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095712 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:41.114250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095714 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095717 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095719 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095724 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095728 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095730 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.095734 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.101102 5051 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.101134 5051 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.101164 5051 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.101169 5051 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101232 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101238 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101242 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101245 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101248 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101250 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101253 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101256 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:41.114724 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101259 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101261 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101264 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101267 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101270 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101273 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101275 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101279 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101284 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101287 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101291 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101294 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101296 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101299 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101302 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101304 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101307 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101309 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101312 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:41.115284 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101315 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101322 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101325 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101328 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101331 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101334 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101337 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101339 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101342 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101344 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101347 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101350 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101352 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101355 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101357 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101360 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101363 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101365 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101368 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:41.116097 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101370 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101373 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101377 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101391 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101395 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101399 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101402 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101405 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101408 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101410 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101413 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101416 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101419 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101422 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101425 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101427 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101430 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101433 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101436 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:41.116890 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101439 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101442 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101444 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101447 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101449 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101452 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101455 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101457 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101460 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101463 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101466 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101469 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101471 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101474 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101483 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101486 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101489 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101491 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101494 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101497 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:41.117714 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101500 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101502 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101505 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101507 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101510 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101513 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101515 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101518 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101520 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101523 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101526 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101528 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101531 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101534 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101537 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101540 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101542 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101545 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101548 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:41.118250 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101550 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101553 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.101581 5051 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101734 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101738 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101741 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101744 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101747 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101750 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101752 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101755 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101758 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101760 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101763 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101766 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101769 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:41.118708 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101771 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101774 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101777 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101779 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101782 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101784 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101787 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101790 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101792 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101795 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101798 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101800 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101803 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101806 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101808 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101811 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101814 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101816 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101819 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101821 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:41.119109 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101824 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101826 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101829 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101831 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101834 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101837 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101840 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101843 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101845 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101848 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101850 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101852 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101855 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101858 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101860 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101864 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101868 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101871 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101874 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:41.119701 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101876 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101881 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101884 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101887 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101889 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101892 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101894 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101897 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101900 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101903 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101905 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101908 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101911 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101913 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101916 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101918 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101921 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101924 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101926 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:41.120211 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101930 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101933 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101935 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101938 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101941 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101943 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101947 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101951 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101954 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101957 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101960 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101962 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101965 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101968 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101971 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101974 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101977 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101979 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101982 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:41.120688 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101984 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101986 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101989 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101992 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101994 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.101997 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102000 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102002 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102005 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102007 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102010 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102012 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102015 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102018 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102020 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102023 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.102046 5051 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 00:57:41.121254 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102080 5051 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102085 5051 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102087 5051 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102090 5051 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102093 5051 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102096 5051 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102098 5051 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102101 5051 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102104 5051 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102106 5051 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102111 5051 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102114 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102137 5051 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102142 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102145 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102148 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102150 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102153 5051 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102155 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102158 5051 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 00:57:41.121973 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102161 5051 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102164 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102166 5051 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102169 5051 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102171 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102174 5051 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102177 5051 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102179 5051 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102182 5051 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102186 5051 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102190 5051 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102194 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102197 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102200 5051 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102202 5051 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102205 5051 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102207 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102211 5051 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102213 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 00:57:41.122706 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102216 5051 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102218 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102221 5051 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102229 5051 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102231 5051 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102234 5051 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102236 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102239 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102242 5051 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102244 5051 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102247 5051 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102249 5051 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102253 5051 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102255 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102258 5051 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102261 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102263 5051 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102266 5051 feature_gate.go:386] unrecognized feature gate: Example Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102268 5051 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 00:57:41.123272 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102272 5051 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102275 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102279 5051 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102281 5051 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102284 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102288 5051 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102291 5051 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102294 5051 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102297 5051 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102300 5051 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102303 5051 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102306 5051 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102309 5051 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102311 5051 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102314 5051 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102316 5051 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102320 5051 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102323 5051 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102325 5051 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102327 5051 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 00:57:41.123758 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102330 5051 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102333 5051 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102335 5051 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102337 5051 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102340 5051 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102343 5051 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102345 5051 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102348 5051 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102351 5051 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102353 5051 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102356 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102359 5051 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102361 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102364 5051 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102366 5051 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102369 5051 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102371 5051 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102374 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102378 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102380 5051 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 00:57:41.124489 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102383 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102385 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102388 5051 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102390 5051 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102393 5051 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102395 5051 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102398 5051 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:41.102400 5051 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.102467 5051 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.103492 5051 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.103550 5051 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.103735 5051 server.go:1014] "Starting client certificate rotation" Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.103874 5051 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.103923 5051 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 00:51:58 +0000 UTC" deadline="2028-06-25 21:55:49.099180602 +0000 UTC" Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.103962 5051 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14396h58m7.995220582s" Nov 04 00:57:41.125162 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.104218 5051 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:41.125686 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.105286 5051 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 00:57:41.125686 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.106508 5051 log.go:25] "Validated CRI v1 runtime API" Nov 04 00:57:41.125686 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.108351 5051 log.go:25] "Validated CRI v1 image API" Nov 04 00:57:41.125686 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.109254 5051 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 00:57:41.125686 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.111481 5051 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 940ded07-229b-49ff-8704-ce7ab1eec3f8:/dev/nvme0n1p4 9afb4279-08e1-43bd-a574-76d8bf9f4e8d:/dev/nvme0n1p3] Nov 04 00:57:41.125902 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.111502 5051 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05fc03a562394dad9dcc7c076bd724c757e3abc030e4aacc92b620e9fcb4de6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05fc03a562394dad9dcc7c076bd724c757e3abc030e4aacc92b620e9fcb4de6e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06235872e24e5f1763a8545a9e07d1730cab3c58732793911f5888e5a007bde9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06235872e24e5f1763a8545a9e07d1730cab3c58732793911f5888e5a007bde9/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a940fb409c7170d592362e00ec0c2beadf3a50c2102d26c10bd1c05dd6dad67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a940fb409c7170d592362e00ec0c2beadf3a50c2102d26c10bd1c05dd6dad67/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bbaee0ae0cd851dba5ede4a2f5c3c7cffc339fd7604199922db1c988f232812/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bbaee0ae0cd851dba5ede4a2f5c3c7cffc339fd7604199922db1c988f232812/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/241c72194a633d194bd11f6c7701f43879c5912719b72bd31b9c38201f356830/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/241c72194a633d194bd11f6c7701f43879c5912719b72bd31b9c38201f356830/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26a695a2da2a162d102bce6a783823e17516262cd423f2298bed353ef5b1bec5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26a695a2da2a162d102bce6a783823e17516262cd423f2298bed353ef5b1bec5/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3df5bb11ab6ab6c7a2827925a728e7a8668d6af074c86473c65e094509680e32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3df5bb11ab6ab6c7a2827925a728e7a8668d6af074c86473c65e094509680e32/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dbb831cc7b9913eb4c21b4a7e7a8d7246d65af7722c61405ed3e6363c7330b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dbb831cc7b9913eb4c21b4a7e7a8d7246d65af7722c61405ed3e6363c7330b4/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50c26ad6cc1cb7efb91a498af2c08443e1e80c83ddcca6f0945f31f6fb955e50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50c26ad6cc1cb7efb91a498af2c08443e1e80c83ddcca6f0945f31f6fb955e50/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5311f6e8245b83b66782847327555239f801dec8e83745900c465a9ddb43afef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5311f6e8245b83b66782847327555239f801dec8e83745900c465a9ddb43afef/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/570379eee1ef601a71e6a50ba4e92747a095999d05d12476a1256eaf47e86abd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/570379eee1ef601a71e6a50ba4e92747a095999d05d12476a1256eaf47e86abd/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57183baeb6c2daf9e4b7d2a79f64a1e6fbbb217c15e01a697909955d5247a5e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57183baeb6c2daf9e4b7d2a79f64a1e6fbbb217c15e01a697909955d5247a5e5/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dafca7abc9eb105558fa198a460f6200041eeaed8974ca72bf3928f5a65d0fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dafca7abc9eb105558fa198a460f6200041eeaed8974ca72bf3928f5a65d0fb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/827a6a1b277083251715d55f9a2bcf3d78f6fc9f06ace303c578bd348ad12944/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/827a6a1b277083251715d55f9a2bcf3d78f6fc9f06ace303c578bd348ad12944/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d1a583db533a1b477e566ec28c586797b3898f1ec1e2c2d68af2bfbb1c58942/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d1a583db533a1b477e566ec28c586797b3898f1ec1e2c2d68af2bfbb1c58942/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/920363e7ada1def36659576aa31abf5fc2cbcef52f5bd8a5eaf6dc80ba6371d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/920363e7ada1def36659576aa31abf5fc2cbcef52f5bd8a5eaf6dc80ba6371d2/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93a338d9cb30e8d5fe70d82c4d8485b1df68152bf391f6d029caa34d4ae5d383/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93a338d9cb30e8d5fe70d82c4d8485b1df68152bf391f6d029caa34d4ae5d383/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab74b5ef9f2a362a896fc842c43c761ced299b84f83cd0d663eb9e1527c433b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab74b5ef9f2a362a896fc842c43c761ced299b84f83cd0d663eb9e1527c433b0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afdc3c7a78e87a2dfa2d67a2961090924b7afe530fe9ecc441f15e941397a578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afdc3c7a78e87a2dfa2d67a2961090924b7afe530fe9ecc441f15e941397a578/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aff40f30cc8a15b11e4219db1c765a0ca2efe27e9cfd477651e18db5871fb28d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aff40f30cc8a15b11e4219db1c765a0ca2efe27e9cfd477651e18db5871fb28d/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc42596f360aaadedf53aa8805a79bd93fdf2374aa451a6fe4460495ea0ef2e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc42596f360aaadedf53aa8805a79bd93fdf2374aa451a6fe4460495ea0ef2e1/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5158317b1a64306381ef738db0f9d6b565351eb291be2943ecd9390e6b2b1f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5158317b1a64306381ef738db0f9d6b565351eb291be2943ecd9390e6b2b1f8/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4fa09327650c9e2e4b2ddfaff4f968d1b740be1c68bf83d0805136a22ede284/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4fa09327650c9e2e4b2ddfaff4f968d1b740be1c68bf83d0805136a22ede284/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5b036939fa5a162d05daba808cab6c16ba1a7d3bc1debd8ed08861578b20104/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5b036939fa5a162d05daba808cab6c16ba1a7d3bc1debd8ed08861578b20104/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f688ab99c75d6ab87c4e41c21a5ea21bbe92b9f815cf6d575a7e673601e1f7ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f688ab99c75d6ab87c4e41c21a5ea21bbe92b9f815cf6d575a7e673601e1f7ae/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8a3af112af34fc6dbb6d93c1b656c2066586e75788a6f61a43fe5acdfbf8062/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8a3af112af34fc6dbb6d93c1b656c2066586e75788a6f61a43fe5acdfbf8062/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04f82b22-3949-44d3-abe9-6473a2f68f76/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/04f82b22-3949-44d3-abe9-6473a2f68f76/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08221c6a-1a73-4922-a769-caeb42c194bb/volumes/kubernetes.io~projected/kube-api-access-gl6xd:{mountpoint:/var/lib/kubelet/pods/08221c6a-1a73-4922-a769-caeb42c194bb/volumes/kubernetes.io~projected/kube-api-access-gl6xd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c8ada62-5c11-4462-a29d-a576d7c162df/volumes/kubernetes.io~projected/kube-api-access-mgbb6:{mountpoint:/var/lib/kubelet/pods/0c8ada62-5c11-4462-a29d-a576d7c162df/volumes/kubernetes.io~projected/kube-api-access-mgbb6 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volumes/kubernetes.io~projected/kube-api-access-75b2w:{mountpoint:/var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volumes/kubernetes.io~projected/kube-api-access-75b2w major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86/volumes/kubernetes.io~projected/kube-api-access-jv5r4:{mountpoint:/var/lib/kubelet/pods/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86/volumes/kubernetes.io~projected/kube-api-access-jv5r4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3391f8ba-2fe6-4950-b644-5361fc8c2e48/volumes/kubernetes.io~projected/kube-api-access-kfhbb:{mountpoint:/var/lib/kubelet/pods/3391f8ba-2fe6-4950-b644-5361fc8c2e48/volumes/kubernetes.io~projected/kube-api-access-kfhbb major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~projected/kube-api-access-dc462:{mountpoint:/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~projected/kube-api-access-dc462 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7/volumes/kubernetes.io~projected/kube-api-access-m95cq:{mountpoint:/var/lib/kubelet/pods/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7/volumes/kubernetes.io~projected/kube-api-access-m95cq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~projected/kube-api-access-6sb2n:{mountpoint:/var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~projected/kube-api-access-6sb2n major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51ff9909-e56c-4d27-be31-574bf33a8f75/volumes/kubernetes.io~projected/kube-api-access-9cjjp:{mountpoint:/var/lib/kubelet/pods/51ff9909-e56c-4d27-be31-574bf33a8f75/volumes/kubernetes.io~projected/kube-api-access-9cjjp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b2c9231-5813-4143-93ca-2457278f37a2/volumes/kubernetes.io~projected/kube-api-access-nlml8:{mountpoint:/var/lib/kubelet/pods/5b2c9231-5813-4143-93ca-2457278f37a2/volumes/kubernetes.io~projected/kube-api-access-nlml8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56/volumes/kubernetes.io~projected/kube-api-access-htxj6:{mountpoint:/var/lib/kubelet/pods/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56/volumes/kubernetes.io~projected/kube-api-access-htxj6 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66c16477-f3db-43cf-9830-39fe0267bdce/volumes/kubernetes.io~projected/kube-api-access-xswcp:{mountpoint:/var/lib/kubelet/pods/66c16477-f3db-43cf-9830-39fe0267bdce/volumes/kubernetes.io~projected/kube-api-access-xswcp major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa9b05c-8730-4590-8709-0275e8be2631/volumes/kubernetes.io~projected/kube-api-access-nwzr6:{mountpoint:/var/lib/kubelet/pods/6aa9b05c-8730-4590-8709-0275e8be2631/volumes/kubernetes.io~projected/kube-api-access-nwzr6 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa9b05c-8730-4590-8709-0275e8be2631/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6aa9b05c-8730-4590-8709-0275e8be2631/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31/volumes/kubernetes.io~secret/agent-certs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/756d0c06-26e9-4c88-8520-34f21fa6a9bd/volumes/kubernetes.io~projected/kube-api-access-w8d5k:{mountpoint:/var/lib/kubelet/pods/756d0c06-26e9-4c88-8520-34f21fa6a9bd/volumes/kubernetes.io~projected/kube-api-access-w8d5k major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/767984a4-597e-4498-a6bf-c988a7b228d1/volumes/kubernetes.io~projected/kube-api-access-gskr9:{mountpoint:/var/lib/kubelet/pods/767984a4-597e-4498-a6bf-c988a7b228d1/volumes/kubernetes.io~projected/kube-api-access-gskr9 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac/volumes/kubernetes.io~projected/kube-api-access-k6b48:{mountpoint:/var/lib/kubelet/pods/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac/volumes/kubernetes.io~projected/kube-api-access-k6b48 major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac/volumes/kubernetes.io~secret/signing-key major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~projected/kube-api-access-72nkw:{mountpoint:/var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~projected/kube-api-access-72nkw major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~secret/default-certificate major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6/volumes/kubernetes.io~projected/kube-api-access-29mnh:{mountpoint:/var/lib/kubelet/pods/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6/volumes/kubernetes.io~projected/kube-api-access-29mnh major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dbd7a56-ab21-413c-abac-36b1f79f452e/volumes/kubernetes.io~projected/kube-api-access-tvpsh:{mountpoint:/var/lib/kubelet/pods/8dbd7a56-ab21-413c-abac-36b1f79f452e/volumes/kubernetes.io~projected/kube-api-access-tvpsh major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dbd7a56-ab21-413c-abac-36b1f79f452e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8dbd7a56-ab21-413c-abac-36b1f79f452e/volumes/kubernetes.io~secret/cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/903f3694-f0a7-4d5f-828b-20bced1b1066/volumes/kubernetes.io~projected/kube-api-access-swhr9:{mountpoint:/var/lib/kubelet/pods/903f3694-f0a7-4d5f-828b-20bced1b1066/volumes/kubernetes.io~projected/kube-api-access-swhr9 major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/969ca7d8-004d-4f22-9929-96099b5f011c/volumes/kubernetes.io~projected/kube-api-access-5rs4s:{mountpoint:/var/lib/kubelet/pods/969ca7d8-004d-4f22-9929-96099b5f011c/volumes/kubernetes.io~projected/kube-api-access-5rs4s major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9747f7d7-39f9-452d-9e9a-dbc9de48c89e/volumes/kubernetes.io~projected/kube-api-access-76kwx:{mountpoint:/var/lib/kubelet/pods/9747f7d7-39f9-452d-9e9a-dbc9de48c89e/volumes/kubernetes.io~projected/kube-api-access-76kwx major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9747f7d7-39f9-452d-9e9a-dbc9de48c89e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9747f7d7-39f9-452d-9e9a-dbc9de48c89e/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac81ee2f-5c6a-4ead-89c5-2a181b928494/volumes/kubernetes.io~projected/kube-api-access-x7l6w:{mountpoint:/var/lib/kubelet/pods/ac81ee2f-5c6a-4ead-89c5-2a181b928494/volumes/kubernetes.io~projected/kube-api-access-x7l6w major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0807c2d-b756-42e8-86b3-97c30f23df72/volumes/kubernetes.io~projected/kube-api-access-5dqxb:{mountpoint:/var/lib/kubelet/pods/b0807c2d-b756-42e8-86b3-97c30f23df72/volumes/kubernetes.io~projected/kube-api-access-5dqxb major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9027aeb-9f01-4ef8-b936-15c9382b2af8/volumes/kubernetes.io~projected/kube-api-access-mnff8:{mountpoint:/var/lib/kubelet/pods/c9027aeb-9f01-4ef8-b936-15c9382b2af8/volumes/kubernetes.io~projected/kube-api-access-mnff8 major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d648b600-369c-47b0-ba02-6b3df1252bed/volumes/kubernetes.io~projected/kube-api-access-n98tj:{mountpoint:/var/lib/kubelet/pods/d648b600-369c-47b0-ba02-6b3df1252bed/volumes/kubernetes.io~projected/kube-api-access-n98tj major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd/volumes/kubernetes.io~projected/kube-api-access-d9x4t:{mountpoint:/var/lib/kubelet/pods/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd/volumes/kubernetes.io~projected/kube-api-access-d9x4t major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~projected/kube-api-access-h7xnz:{mountpoint:/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~projected/kube-api-access-h7xnz major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7f2ccca-45df-41f8-bbf9-0b8aaa791160/volumes/kubernetes.io~projected/kube-api-access-hk25v:{mountpoint:/var/lib/kubelet/pods/e7f2ccca-45df-41f8-bbf9-0b8aaa791160/volumes/kubernetes.io~projected/kube-api-access-hk25v major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ede327f8-e1cf-4596-abe5-de379f528b15/volumes/kubernetes.io~projected/kube-api-access-xlnpx:{mountpoint:/var/lib/kubelet/pods/ede327f8-e1cf-4596-abe5-de379f528b15/volumes/kubernetes.io~projected/kube-api-access-xlnpx major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ede327f8-e1cf-4596-abe5-de379f528b15/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ede327f8-e1cf-4596-abe5-de379f528b15/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4bef706-ca6f-4260-962f-ad0599ad093a/volumes/kubernetes.io~projected/kube-api-access-jddfl:{mountpoint:/var/lib/kubelet/pods/f4bef706-ca6f-4260-962f-ad0599ad093a/volumes/kubernetes.io~projected/kube-api-access-jddfl major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/bc78a3d6cb11840179a46ae3d7db3b9b03c6ccf923ebd061167e71c5998413da/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b8c9a6e661a4edc14c8fe354d62f7ccceecc3515f719c41e3fa26692293e7295/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/da28f4e3cc15632a8158969ca27d7cfba91dff1bbc858d6dac2300588b666e61/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d39058c6fad8b995580199a8e879530d66cb848f39233fbddb7f59c668b0da19/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b14d6b74d52414adfbd8ecbcc39b7f6289157f146c0f39fe9bd157a4d3ce39f1/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2f41fd5bc0cba31bf6060fb6a83d32c1bf4c0c878622877aa8a67e191d484a2c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6fee4ed0a25eb99add91f9851de524721254f429fb5583aaf9dc621951db2f16/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3ef14008c697ada9ab622d638ed879226ce030cd186ed170b816874fd6dcb24c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d335fb7772e34421571a0c41fdf2e881cee08e2dd151a45562413a781e41628c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/4dbb4e4cd296ddcca77c37af0239ca369578858146293bf09d0fa3f4841dfa2f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/6b0c11ac67c577fe7bd312b3ce92f151f9135aa3633500b39046567c12af7350/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/8af0fb34bb1be72369168a5f54082261a7a65afa47b088fa4f68bd014f19924c/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f78f3203a50c7fe8f081f991138533ff2083d92e680ddf8e87983218ee350f25/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c8a14e39b10560b304e5457b9d0dba4134098470f785329495b0b4e806f8300a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4acecda48e5c1c6622df404a6a46988830fc2d2936d8e1eaa02278ee90d0ce00/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0ba7a1b7e936f670a8020feeb8f31ab8328251c34f99eec7343d198133f66536/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8a319a31f7b547013f8b88c5543920263f4a38060b02351f02d878310073c4e0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/efc0a24e99c09017eb64f739a61c1911f45e57435b5e2b0c4572f3d1cda3f4c2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/41e16a44ca578a6ab6fd433cac8e385463537b44fcbd41c3fb9f6490de20bf20/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/75a6c03df53de7293f7145b01064295028d4359a34dee16732942dd6bfe439e8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e2331406fa845492babc705e85095fc02150664cc03f09f811791653ecc7721d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/6a65bb6d6c5e97bb010cd5f5a104a5563c67c442f8cc89ab04669f2562e2f000/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/f6f75b249ec75da1473fad98ef64bba181ab7084ee39d77513ca8600ea9eb6fb/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/6b9ac994ffdfb87a1da4f77de2eec8d912dd8a74ca2a2de99af5b17c4d524cab/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/eedce16e9f910f7f1da2a0b3c62076c9a6d561f4e6478f651de7fe30fe7f7e18/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/5c740f6246f3f12d9419795a53384d2a56c4613ee809ea8cc2524b48a37bbba9/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/60b56d9a19edf9d572849ecb822f0f13f4dab2016b77e707283b1e35ebc7b973/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/56c8dddc1001c159b957bbd1a2b7ce1836c57867ba058cb63b1a75382f3ce30d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d7ff18e37acf570c04ffbe969198b08bb2da391cbbc9dc1b52908b3674bba25b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/ea63e55e922f96c511c7f8abdbdd1a5421918466e4c2a0bc5236b42a406ccf4a/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/3b2b80b88cb2a3a408dfd5d47c9eda7c0bf688047c42fb0bb590733afb62a909/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/a6518aab729da4f803abee9464e8420ac27b0b621aeccb179ea12682fcfd022e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/551417887f5c225d4793ef6c9642bbe605391dbb2b3ee0b51ca4b8b448e51688/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/664c1bcdb7f784f26333f94bfa4d3acd8cbd7d83e3bf43f3eb79d7d60acedfee/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/a618b9d1556c927064d6cd38b331f0c529f5fbf7a13f602b8395f7e3db26cf52/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/629c4f6c80a4cdb44876e5c93013485fef2a78d2948c9f9e8c95b7dd3917ce03/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/fd86c240825303241338e5e569cd2b93f2308a5f0ba63dd78b5c2543ca24e6f4/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/13c9d9c5b65cab86dd8063b1369a40726ca248ccaa440ba4c38b84d57f19e99f/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/0b4ae8b84419a2c673f19b143edb5221cf79eb703c629415d8140b3f540236bc/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/89750c3f87f05b809e190c37e2a7f65e17b60a9a032f28dd429ef15654fc09d3/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/01cb4ad3318a4b8b5f4a56e5ef09e5410307b1710762519ef8abc10d3ea7ca9f/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/4b1f0237e1f0ea5a9d2b770a8e8093a4ad04239c18c6d04788011620be7cf874/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/333a18fc24c91407175e9723c0039c72f55e2a699662f39be99c226a80347454/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/a08868f1d82b855626209031ebd0f752897c93848851cb81f646cfd53bf50e3e/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/0cdd721ebd7905022616e7ed5fdd6d4d8d168ae8d71eb4cf4fa57aec62dbe704/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/fcb118f0e720f848e2db86f911d393b8a1966b10764582755328973afa1fca12/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/f16315c199789e0e684cd92564deb9521082a134b486f9fb788ca9aeb0bc5064/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/f8ed7f8dfd1f101f776b9d91ac8b4827f11cca3d8ad9fca38b49d056b7b927b1/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/7faa2b9ef1d68d25557cc18c8911f4e0049e39604342cdc35884a0368c3d0ba7/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/c9985fea00244a96f5541d9878f98ef17e9be594e22a6325ff365f2388201e4e/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/dde1e658d81a1b7730ad053785c55ca37e2db933aee9f17025c2c49a2dffde01/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/e3fdc5a063d1c7c1b78342257a6ea20cada37b636a956d0b0bbf8347675ee272/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/12fe542cd7f2eb5b67edd7254e89030a670dacf06b3c25d4ed76fda95c2ee918/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b0cdf0206cc7a5c4ed130b8e4f96c39e26188549e99a38c4c80799cdbe9052db/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/dd5ea8962146382b2856fe5f7e0205520911c2d5eae316fd8bc5fb968163cfdd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1eec7f2924accaf25d72810b2a1d85cad2a7a9d4d5c61a7ce74d830f6c5dc07d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b668166b1837371a445d6206bb7d2ee62aaba19c6b2e43ae0b3d046e85d414d4/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/62dff4f88877cedab91d8f2d3353cc6158bac28a91e53babbdcbe93449935248/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/55a57e48296dca411c8b6d85a0ea45fd57bb520fbfe93b17296e96608b9c7927/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d2da8cd75655de9d657414b1f921aaba4378560e5cbd4507ce0be1c752cb85f6/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/035885664c65a0063c18baa284d0dbb56389d9095d7edf5eee01780f2ae020e4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b595535c4c786c3ff7498e93513ac31bfe5039db777247a058009e48d96d35c8/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.122453 5051 manager.go:217] Machine: {Timestamp:2026-11-04 00:57:41.121972694 +0000 UTC m=+0.093721556 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100433 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b83d2d941938a9da675ee145c52cd SystemUUID:ec2b83d2-d941-938a-9da6-75ee145c52cd BootID:6f8aa927-c3a9-41e1-bdb7-6c61df5001d0 Filesystems:[{Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa9b05c-8730-4590-8709-0275e8be2631/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc42596f360aaadedf53aa8805a79bd93fdf2374aa451a6fe4460495ea0ef2e1/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9747f7d7-39f9-452d-9e9a-dbc9de48c89e/volumes/kubernetes.io~projected/kube-api-access-76kwx DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aff40f30cc8a15b11e4219db1c765a0ca2efe27e9cfd477651e18db5871fb28d/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7f2ccca-45df-41f8-bbf9-0b8aaa791160/volumes/kubernetes.io~projected/kube-api-access-hk25v DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5b036939fa5a162d05daba808cab6c16ba1a7d3bc1debd8ed08861578b20104/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8dbd7a56-ab21-413c-abac-36b1f79f452e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/920363e7ada1def36659576aa31abf5fc2cbcef52f5bd8a5eaf6dc80ba6371d2/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51ff9909-e56c-4d27-be31-574bf33a8f75/volumes/kubernetes.io~projected/kube-api-access-9cjjp DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ede327f8-e1cf-4596-abe5-de379f528b15/volumes/kubernetes.io~projected/kube-api-access-xlnpx DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bbaee0ae0cd851dba5ede4a2f5c3c7cffc339fd7604199922db1c988f232812/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volumes/kubernetes.io~projected/kube-api-access-75b2w DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab74b5ef9f2a362a896fc842c43c761ced299b84f83cd0d663eb9e1527c433b0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afdc3c7a78e87a2dfa2d67a2961090924b7afe530fe9ecc441f15e941397a578/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ede327f8-e1cf-4596-abe5-de379f528b15/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd/volumes/kubernetes.io~projected/kube-api-access-d9x4t DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/827a6a1b277083251715d55f9a2bcf3d78f6fc9f06ace303c578bd348ad12944/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/241c72194a633d194bd11f6c7701f43879c5912719b72bd31b9c38201f356830/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/969ca7d8-004d-4f22-9929-96099b5f011c/volumes/kubernetes.io~projected/kube-api-access-5rs4s DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~projected/kube-api-access-72nkw DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/903f3694-f0a7-4d5f-828b-20bced1b1066/volumes/kubernetes.io~projected/kube-api-access-swhr9 DeviceMajor:0 DeviceMinor:322 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:350 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5158317b1a64306381ef738db0f9d6b565351eb291be2943ecd9390e6b2b1f8/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4bef706-ca6f-4260-962f-ad0599ad093a/volumes/kubernetes.io~projected/kube-api-access-jddfl DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b2c9231-5813-4143-93ca-2457278f37a2/volumes/kubernetes.io~projected/kube-api-access-nlml8 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4fa09327650c9e2e4b2ddfaff4f968d1b740be1c68bf83d0805136a22ede284/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05fc03a562394dad9dcc7c076bd724c757e3abc030e4aacc92b620e9fcb4de6e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~projected/kube-api-access-dc462 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d1a583db533a1b477e566ec28c586797b3898f1ec1e2c2d68af2bfbb1c58942/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9747f7d7-39f9-452d-9e9a-dbc9de48c89e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7/volumes/kubernetes.io~projected/kube-api-access-m95cq DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86/volumes/kubernetes.io~projected/kube-api-access-jv5r4 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa9b05c-8730-4590-8709-0275e8be2631/volumes/kubernetes.io~projected/kube-api-access-nwzr6 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50c26ad6cc1cb7efb91a498af2c08443e1e80c83ddcca6f0945f31f6fb955e50/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0c8ada62-5c11-4462-a29d-a576d7c162df/volumes/kubernetes.io~projected/kube-api-access-mgbb6 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08221c6a-1a73-4922-a769-caeb42c194bb/volumes/kubernetes.io~projected/kube-api-access-gl6xd DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/767984a4-597e-4498-a6bf-c988a7b228d1/volumes/kubernetes.io~projected/kube-api-access-gskr9 DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26a695a2da2a162d102bce6a783823e17516262cd423f2298bed353ef5b1bec5/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8a3af112af34fc6dbb6d93c1b656c2066586e75788a6f61a43fe5acdfbf8062/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6/volumes/kubernetes.io~projected/kube-api-access-29mnh DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/04f82b22-3949-44d3-abe9-6473a2f68f76/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9027aeb-9f01-4ef8-b936-15c9382b2af8/volumes/kubernetes.io~projected/kube-api-access-mnff8 DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d648b600-369c-47b0-ba02-6b3df1252bed/volumes/kubernetes.io~projected/kube-api-access-n98tj DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b0807c2d-b756-42e8-86b3-97c30f23df72/volumes/kubernetes.io~projected/kube-api-access-5dqxb DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51fbfff7-c90b-4726-bbff-6ebbcc943626/volumes/kubernetes.io~projected/kube-api-access-6sb2n DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93a338d9cb30e8d5fe70d82c4d8485b1df68152bf391f6d029caa34d4ae5d383/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac/volumes/kubernetes.io~projected/kube-api-access-k6b48 DeviceMajor:0 DeviceMinor:355 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1006149a-92ee-43c1-bb41-6aaf1df41c36/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8dbd7a56-ab21-413c-abac-36b1f79f452e/volumes/kubernetes.io~projected/kube-api-access-tvpsh DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57183baeb6c2daf9e4b7d2a79f64a1e6fbbb217c15e01a697909955d5247a5e5/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66c16477-f3db-43cf-9830-39fe0267bdce/volumes/kubernetes.io~projected/kube-api-access-xswcp DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3df5bb11ab6ab6c7a2827925a728e7a8668d6af074c86473c65e094509680e32/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/756d0c06-26e9-4c88-8520-34f21fa6a9bd/volumes/kubernetes.io~projected/kube-api-access-w8d5k DeviceMajor:0 DeviceMinor:321 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a940fb409c7170d592362e00ec0c2beadf3a50c2102d26c10bd1c05dd6dad67/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dbb831cc7b9913eb4c21b4a7e7a8d7246d65af7722c61405ed3e6363c7330b4/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56/volumes/kubernetes.io~projected/kube-api-access-htxj6 DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d024224-166d-4cb4-976f-75e0b47addae/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3391f8ba-2fe6-4950-b644-5361fc8c2e48/volumes/kubernetes.io~projected/kube-api-access-kfhbb DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/570379eee1ef601a71e6a50ba4e92747a095999d05d12476a1256eaf47e86abd/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dafca7abc9eb105558fa198a460f6200041eeaed8974ca72bf3928f5a65d0fb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f688ab99c75d6ab87c4e41c21a5ea21bbe92b9f815cf6d575a7e673601e1f7ae/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06235872e24e5f1763a8545a9e07d1730cab3c58732793911f5888e5a007bde9/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac81ee2f-5c6a-4ead-89c5-2a181b928494/volumes/kubernetes.io~projected/kube-api-access-x7l6w DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes/kubernetes.io~projected/kube-api-access-h7xnz DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5311f6e8245b83b66782847327555239f801dec8e83745900c465a9ddb43afef/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a940fb409c7170 MacAddress:ce:68:4a:2a:91:ff Speed:10000 Mtu:8901} {Name:241c72194a633d1 MacAddress:7a:b4:cc:e5:ae:75 Speed:10000 Mtu:8901} {Name:26a695a2da2a162 MacAddress:d6:e4:da:d0:5d:a5 Speed:10000 Mtu:8901} {Name:3df5bb11ab6ab6c MacAddress:aa:d9:38:0b:04:be Speed:10000 Mtu:8901} {Name:4dbb831cc7b9913 MacAddress:12:10:a0:b4:85:c6 Speed:10000 Mtu:8901} {Name:50c26ad6cc1cb7e MacAddress:02:11:23:56:1d:51 Speed:10000 Mtu:8901} {Name:5311f6e8245b83b MacAddress:9e:ee:fb:59:92:72 Speed:10000 Mtu:8901} {Name:57183baeb6c2daf MacAddress:6e:90:af:f0:32:2d Speed:10000 Mtu:8901} {Name:827a6a1b2770832 MacAddress:0e:be:22:75:ac:6d Speed:10000 Mtu:8901} {Name:920363e7ada1def MacAddress:d2:77:83:24:fe:e3 Speed:10000 Mtu:8901} {Name:afdc3c7a78e87a2 MacAddress:da:a2:f6:c2:7c:01 Speed:10000 Mtu:8901} {Name:aff40f30cc8a15b MacAddress:62:95:f8:cc:f9:12 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e4:a8:74:fd Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:4d:22:f6:0d:07 Speed:0 Mtu:8901} {Name:d5158317b1a6430 MacAddress:de:64:d0:9e:45:ff Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e4:a8:74:fd Speed:0 Mtu:9001} {Name:f4fa09327650c9e MacAddress:56:ac:c4:d1:40:97 Speed:10000 Mtu:8901} {Name:f688ab99c75d6ab MacAddress:f6:ca:99:18:37:ec Speed:10000 Mtu:8901} {Name:f8a3af112af34fc MacAddress:12:b9:a6:d5:13:70 Speed:10000 Mtu:8901} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:88:c8:e0:e8:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.122873 5051 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.122923 5051 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123307 5051 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123332 5051 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-48.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123478 5051 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123485 5051 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123497 5051 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123517 5051 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123537 5051 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123731 5051 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123837 5051 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123872 5051 kubelet.go:493] "Attempting to sync node with API server" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123882 5051 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123891 5051 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123900 5051 kubelet.go:405] "Adding apiserver pod source" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.123913 5051 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.125138 5051 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.125151 5051 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:41.126209 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.125767 5051 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 00:57:41.128528 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.128513 5051 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 00:57:41.128528 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.128526 5051 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 00:57:41.128635 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.128625 5051 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 00:57:41.128844 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.128833 5051 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 00:57:41.128873 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.128863 5051 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 00:57:41.128974 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.128963 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 00:57:41.129003 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.128987 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 00:57:41.129030 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129002 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 00:57:41.129030 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129018 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 00:57:41.129082 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129034 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 00:57:41.129082 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129045 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 00:57:41.129082 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129061 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 00:57:41.129082 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129074 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 00:57:41.129210 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129088 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 00:57:41.129210 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129097 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 00:57:41.129210 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129114 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 00:57:41.129210 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129138 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 00:57:41.129210 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129194 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 00:57:41.129210 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.129203 5051 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 00:57:41.133286 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.133270 5051 server.go:1263] "Started kubelet" Nov 04 00:57:41.134029 ip-10-0-137-48 systemd[1]: Started Kubernetes Kubelet. Nov 04 00:57:41.141350 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.133422 5051 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 00:57:41.141350 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.133415 5051 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 00:57:41.141350 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.138752 5051 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 00:57:41.141350 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.139985 5051 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 00:57:41.141350 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.140997 5051 server.go:317] "Adding debug handlers to kubelet server" Nov 04 00:57:41.143318 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.143299 5051 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 00:57:41.143417 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.143372 5051 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 00:57:41.143634 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.143597 5051 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 00:52:00 +0000 UTC" deadline="2028-08-06 10:22:10.443166406 +0000 UTC" Nov 04 00:57:41.143634 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.143631 5051 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15393h24m29.299538117s" Nov 04 00:57:41.143810 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.143694 5051 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:41.143868 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.143854 5051 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 00:57:41.143913 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.143869 5051 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 00:57:41.143989 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.143974 5051 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 00:57:41.145362 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.145341 5051 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:41.145981 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.145963 5051 factory.go:55] Registering systemd factory Nov 04 00:57:41.145981 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.145980 5051 factory.go:223] Registration of the systemd container factory successfully Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.146486 5051 factory.go:153] Registering CRI-O factory Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.146505 5051 factory.go:223] Registration of the crio container factory successfully Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.146559 5051 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.146579 5051 factory.go:103] Registering Raw factory Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.146592 5051 manager.go:1199] Started watching for new ooms in manager Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.146970 5051 manager.go:319] Starting recovery of all containers Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147081 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1006149a-92ee-43c1-bb41-6aaf1df41c36" volumeName="kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-config" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147114 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1006149a-92ee-43c1-bb41-6aaf1df41c36" volumeName="kubernetes.io/secret/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147145 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" volumeName="kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-trusted-ca" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147157 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd" volumeName="kubernetes.io/projected/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd-kube-api-access-d9x4t" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147171 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54204af-fcd5-442e-9009-f58ba803f3c4" volumeName="kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147183 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f2ccca-45df-41f8-bbf9-0b8aaa791160" volumeName="kubernetes.io/empty-dir/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-data-volume" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147195 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ede327f8-e1cf-4596-abe5-de379f528b15" volumeName="kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-snapshots" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147206 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1006149a-92ee-43c1-bb41-6aaf1df41c36" volumeName="kubernetes.io/projected/1006149a-92ee-43c1-bb41-6aaf1df41c36-kube-api-access-75b2w" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147221 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08221c6a-1a73-4922-a769-caeb42c194bb" volumeName="kubernetes.io/projected/08221c6a-1a73-4922-a769-caeb42c194bb-kube-api-access-gl6xd" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147234 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3391f8ba-2fe6-4950-b644-5361fc8c2e48" volumeName="kubernetes.io/projected/3391f8ba-2fe6-4950-b644-5361fc8c2e48-kube-api-access-kfhbb" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147246 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa9b05c-8730-4590-8709-0275e8be2631" volumeName="kubernetes.io/projected/6aa9b05c-8730-4590-8709-0275e8be2631-kube-api-access-nwzr6" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147258 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" volumeName="kubernetes.io/projected/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-kube-api-access-76kwx" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147270 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648b600-369c-47b0-ba02-6b3df1252bed" volumeName="kubernetes.io/configmap/d648b600-369c-47b0-ba02-6b3df1252bed-telemetry-config" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147280 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ede327f8-e1cf-4596-abe5-de379f528b15" volumeName="kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-tmp" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147304 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7cc6df-efb4-4ed5-a299-99d3eafd8e56" volumeName="kubernetes.io/configmap/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-serviceca" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147316 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf9dde4-e870-49c8-a34d-1c0a4c85acb6" volumeName="kubernetes.io/projected/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-kube-api-access-29mnh" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147328 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54204af-fcd5-442e-9009-f58ba803f3c4" volumeName="kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147339 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8ada62-5c11-4462-a29d-a576d7c162df" volumeName="kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-binary-copy" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147351 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ff9909-e56c-4d27-be31-574bf33a8f75" volumeName="kubernetes.io/empty-dir/51ff9909-e56c-4d27-be31-574bf33a8f75-tmp-dir" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147371 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa9b05c-8730-4590-8709-0275e8be2631" volumeName="kubernetes.io/secret/6aa9b05c-8730-4590-8709-0275e8be2631-serving-cert" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147391 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9027aeb-9f01-4ef8-b936-15c9382b2af8" volumeName="kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147405 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d648b600-369c-47b0-ba02-6b3df1252bed" volumeName="kubernetes.io/projected/d648b600-369c-47b0-ba02-6b3df1252bed-kube-api-access-n98tj" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147438 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51fbfff7-c90b-4726-bbff-6ebbcc943626" volumeName="kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-tmp" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147450 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c16477-f3db-43cf-9830-39fe0267bdce" volumeName="kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147462 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac" volumeName="kubernetes.io/secret/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-key" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147473 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac81ee2f-5c6a-4ead-89c5-2a181b928494" volumeName="kubernetes.io/projected/ac81ee2f-5c6a-4ead-89c5-2a181b928494-kube-api-access-x7l6w" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147486 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ede327f8-e1cf-4596-abe5-de379f528b15" volumeName="kubernetes.io/projected/ede327f8-e1cf-4596-abe5-de379f528b15-kube-api-access-xlnpx" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147507 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7" volumeName="kubernetes.io/projected/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kube-api-access-m95cq" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147525 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8ada62-5c11-4462-a29d-a576d7c162df" volumeName="kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147538 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f2b307f-e5f0-4116-b2cd-2bce47ae5b86" volumeName="kubernetes.io/secret/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-serving-cert" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147552 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b2c9231-5813-4143-93ca-2457278f37a2" volumeName="kubernetes.io/projected/5b2c9231-5813-4143-93ca-2457278f37a2-kube-api-access-nlml8" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147564 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac" volumeName="kubernetes.io/projected/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-kube-api-access-k6b48" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147576 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54204af-fcd5-442e-9009-f58ba803f3c4" volumeName="kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147588 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7f2ccca-45df-41f8-bbf9-0b8aaa791160" volumeName="kubernetes.io/projected/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-api-access-hk25v" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147606 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603f9b4-53ae-4e45-832f-e5c29ca743b9" volumeName="kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147618 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08de9-97a2-4c5b-ae3a-66a9cd783a31" volumeName="kubernetes.io/configmap/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-konnectivity-ca" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147633 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="767984a4-597e-4498-a6bf-c988a7b228d1" volumeName="kubernetes.io/projected/767984a4-597e-4498-a6bf-c988a7b228d1-kube-api-access-gskr9" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147641 5051 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147645 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" volumeName="kubernetes.io/secret/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-serving-cert" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147662 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8ada62-5c11-4462-a29d-a576d7c162df" volumeName="kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147675 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f2b307f-e5f0-4116-b2cd-2bce47ae5b86" volumeName="kubernetes.io/configmap/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-config" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147688 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="756d0c06-26e9-4c88-8520-34f21fa6a9bd" volumeName="kubernetes.io/projected/756d0c06-26e9-4c88-8520-34f21fa6a9bd-kube-api-access-w8d5k" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147706 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac" volumeName="kubernetes.io/configmap/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-cabundle" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147720 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cf9dde4-e870-49c8-a34d-1c0a4c85acb6" volumeName="kubernetes.io/configmap/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-config" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147731 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dbd7a56-ab21-413c-abac-36b1f79f452e" volumeName="kubernetes.io/projected/8dbd7a56-ab21-413c-abac-36b1f79f452e-kube-api-access-tvpsh" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147743 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="903f3694-f0a7-4d5f-828b-20bced1b1066" volumeName="kubernetes.io/projected/903f3694-f0a7-4d5f-828b-20bced1b1066-kube-api-access-swhr9" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147757 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9027aeb-9f01-4ef8-b936-15c9382b2af8" volumeName="kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147773 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3391f8ba-2fe6-4950-b644-5361fc8c2e48" volumeName="kubernetes.io/configmap/3391f8ba-2fe6-4950-b644-5361fc8c2e48-iptables-alerter-script" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147786 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603f9b4-53ae-4e45-832f-e5c29ca743b9" volumeName="kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147798 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ef08de9-97a2-4c5b-ae3a-66a9cd783a31" volumeName="kubernetes.io/secret/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-agent-certs" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147811 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="969ca7d8-004d-4f22-9929-96099b5f011c" volumeName="kubernetes.io/projected/969ca7d8-004d-4f22-9929-96099b5f011c-kube-api-access-5rs4s" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147825 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0807c2d-b756-42e8-86b3-97c30f23df72" volumeName="kubernetes.io/empty-dir/b0807c2d-b756-42e8-86b3-97c30f23df72-tmp-dir" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147836 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0807c2d-b756-42e8-86b3-97c30f23df72" volumeName="kubernetes.io/projected/b0807c2d-b756-42e8-86b3-97c30f23df72-kube-api-access-5dqxb" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147860 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54204af-fcd5-442e-9009-f58ba803f3c4" volumeName="kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147874 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4bef706-ca6f-4260-962f-ad0599ad093a" volumeName="kubernetes.io/projected/f4bef706-ca6f-4260-962f-ad0599ad093a-kube-api-access-jddfl" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147886 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04f82b22-3949-44d3-abe9-6473a2f68f76" volumeName="kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147899 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603f9b4-53ae-4e45-832f-e5c29ca743b9" volumeName="kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147937 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d024224-166d-4cb4-976f-75e0b47addae" volumeName="kubernetes.io/projected/7d024224-166d-4cb4-976f-75e0b47addae-kube-api-access-72nkw" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147951 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603f9b4-53ae-4e45-832f-e5c29ca743b9" volumeName="kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147969 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9027aeb-9f01-4ef8-b936-15c9382b2af8" volumeName="kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.147984 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1006149a-92ee-43c1-bb41-6aaf1df41c36" volumeName="kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-env-overrides" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148003 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08221c6a-1a73-4922-a769-caeb42c194bb" volumeName="kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-cni-binary-copy" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148019 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08221c6a-1a73-4922-a769-caeb42c194bb" volumeName="kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-multus-daemon-config" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148033 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c8ada62-5c11-4462-a29d-a576d7c162df" volumeName="kubernetes.io/projected/0c8ada62-5c11-4462-a29d-a576d7c162df-kube-api-access-mgbb6" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148044 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d024224-166d-4cb4-976f-75e0b47addae" volumeName="kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-stats-auth" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148057 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dbd7a56-ab21-413c-abac-36b1f79f452e" volumeName="kubernetes.io/secret/8dbd7a56-ab21-413c-abac-36b1f79f452e-cert" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148071 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51fbfff7-c90b-4726-bbff-6ebbcc943626" volumeName="kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-tuned" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148083 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51ff9909-e56c-4d27-be31-574bf33a8f75" volumeName="kubernetes.io/projected/51ff9909-e56c-4d27-be31-574bf33a8f75-kube-api-access-9cjjp" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148099 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b7cc6df-efb4-4ed5-a299-99d3eafd8e56" volumeName="kubernetes.io/projected/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-kube-api-access-htxj6" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148145 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ede327f8-e1cf-4596-abe5-de379f528b15" volumeName="kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-service-ca-bundle" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148159 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603f9b4-53ae-4e45-832f-e5c29ca743b9" volumeName="kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148182 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603f9b4-53ae-4e45-832f-e5c29ca743b9" volumeName="kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148205 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54204af-fcd5-442e-9009-f58ba803f3c4" volumeName="kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148222 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f2b307f-e5f0-4116-b2cd-2bce47ae5b86" volumeName="kubernetes.io/projected/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-kube-api-access-jv5r4" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148235 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3603f9b4-53ae-4e45-832f-e5c29ca743b9" volumeName="kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148250 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51fbfff7-c90b-4726-bbff-6ebbcc943626" volumeName="kubernetes.io/projected/51fbfff7-c90b-4726-bbff-6ebbcc943626-kube-api-access-6sb2n" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148266 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa9b05c-8730-4590-8709-0275e8be2631" volumeName="kubernetes.io/configmap/6aa9b05c-8730-4590-8709-0275e8be2631-config" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148306 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d024224-166d-4cb4-976f-75e0b47addae" volumeName="kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-default-certificate" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148321 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ede327f8-e1cf-4596-abe5-de379f528b15" volumeName="kubernetes.io/secret/ede327f8-e1cf-4596-abe5-de379f528b15-serving-cert" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148336 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="261f5fa6-5956-4a5d-a104-982dc13aed39" volumeName="kubernetes.io/configmap/261f5fa6-5956-4a5d-a104-982dc13aed39-nginx-conf" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148353 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" volumeName="kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-config" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148366 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0807c2d-b756-42e8-86b3-97c30f23df72" volumeName="kubernetes.io/configmap/b0807c2d-b756-42e8-86b3-97c30f23df72-config-volume" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148379 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9027aeb-9f01-4ef8-b936-15c9382b2af8" volumeName="kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148393 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54204af-fcd5-442e-9009-f58ba803f3c4" volumeName="kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148406 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54204af-fcd5-442e-9009-f58ba803f3c4" volumeName="kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148442 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ede327f8-e1cf-4596-abe5-de379f528b15" volumeName="kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-trusted-ca-bundle" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148465 5051 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1006149a-92ee-43c1-bb41-6aaf1df41c36" volumeName="kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-script-lib" seLinuxMountContext="" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148479 5051 reconstruct.go:97] "Volume reconstruction finished" Nov 04 00:57:41.149612 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.148488 5051 reconciler.go:29] "Reconciler: start to sync state" Nov 04 00:57:41.153666 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.151866 5051 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 00:57:41.165851 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.165821 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 00:57:41.166003 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.165885 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 00:57:41.178059 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.178007 5051 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 00:57:41.179939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.179908 5051 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 00:57:41.179939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.179935 5051 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 00:57:41.180110 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.179958 5051 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 00:57:41.180110 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:41.180002 5051 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 00:57:41.184921 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.184902 5051 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 00:57:41.185576 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.185556 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 00:57:41.185709 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.185588 5051 generic.go:356] "Generic (PLEG): container finished" podID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" containerID="8b1ac2c33f7a81e7550c8cc5e675d9803e8f2307b5e6c85fbbd6e1a0817c6094" exitCode=255 Nov 04 00:57:41.190991 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.190965 5051 generic.go:356] "Generic (PLEG): container finished" podID="1a84db2d45c5cb12625222b3483c125f" containerID="b73c9d32e77a52cd15af814a31b9c0212466de5ef3bef800276c9b597fcecc29" exitCode=0 Nov 04 00:57:41.202368 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.202345 5051 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="afa41aa91dba70cd6b5650ec75317bae7e12811f5876d6809a2985e218368365" exitCode=0 Nov 04 00:57:41.202368 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.202363 5051 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="ea3003a9de0b30293c93928e29dfd73b997c3966703033d2a227cb5caf5c6b3e" exitCode=0 Nov 04 00:57:41.202368 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.202369 5051 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="b036c256f474e00da915d93ffcdf96bb7b1da4019e3e9d3a8c8f8a1b6428e49a" exitCode=0 Nov 04 00:57:41.202557 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.202374 5051 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="e62ba671b9e105de2d8dd37a0a87a364a50379722ab93dc37cb62bf93189cd17" exitCode=0 Nov 04 00:57:41.202557 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.202379 5051 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="cbedca9ef1feebccc002dbaf4d819756a55225e290ab7a2cd50ecc71799ced10" exitCode=0 Nov 04 00:57:41.202557 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.202385 5051 generic.go:356] "Generic (PLEG): container finished" podID="0c8ada62-5c11-4462-a29d-a576d7c162df" containerID="0c6c11b4ec6a9ba9445feab09cbe71d6b8a4fcd23ab1d9a4495f1fda87eb21c4" exitCode=0 Nov 04 00:57:41.247726 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.247694 5051 manager.go:324] Recovery completed Nov 04 00:57:41.268229 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268204 5051 cpu_manager.go:226] "Starting" policy="none" Nov 04 00:57:41.268229 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268224 5051 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 00:57:41.268410 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268249 5051 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 00:57:41.268451 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268410 5051 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 00:57:41.268451 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268426 5051 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 00:57:41.268541 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268530 5051 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 00:57:41.268572 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268542 5051 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 00:57:41.268572 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268548 5051 policy_none.go:50] "Start" Nov 04 00:57:41.268572 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268562 5051 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 00:57:41.268649 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268575 5051 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 00:57:41.268722 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268713 5051 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 00:57:41.268752 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268723 5051 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 00:57:41.268752 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.268729 5051 policy_none.go:44] "Start" Nov 04 00:57:41.274883 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.274851 5051 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 00:57:41.277605 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.277589 5051 manager.go:341] "Starting Device Plugin manager" Nov 04 00:57:41.277717 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:41.277622 5051 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 00:57:41.277717 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.277636 5051 server.go:87] "Starting device plugin registration server" Nov 04 00:57:41.278415 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.278397 5051 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 00:57:41.278489 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.278415 5051 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 00:57:41.278553 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.278505 5051 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 00:57:41.278658 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.278623 5051 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 00:57:41.278714 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.278670 5051 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 00:57:41.278714 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.278677 5051 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 00:57:41.278805 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.278729 5051 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T00:57:41.278657307Z","UUID":"5fe91814-22ef-4da4-a3cc-51b5f3668135","Handler":null,"Name":"","Endpoint":""} Nov 04 00:57:41.280094 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.280057 5051 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal"] Nov 04 00:57:41.280191 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.280081 5051 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 00:57:41.280191 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.280174 5051 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 00:57:41.280296 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:41.280253 5051 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 00:57:41.280992 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.280952 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"1a84db2d45c5cb12625222b3483c125f","Type":"ContainerStarted","Data":"3d6f8fa1163a57e94ce746ea37b9da17d2892244215162f8bec361a7959d5c3a"} Nov 04 00:57:41.281087 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.281004 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"1a84db2d45c5cb12625222b3483c125f","Type":"ContainerDied","Data":"b73c9d32e77a52cd15af814a31b9c0212466de5ef3bef800276c9b597fcecc29"} Nov 04 00:57:41.281087 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.281067 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"1a84db2d45c5cb12625222b3483c125f","Type":"ContainerStarted","Data":"8d1a583db533a1b477e566ec28c586797b3898f1ec1e2c2d68af2bfbb1c58942"} Nov 04 00:57:41.281257 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.281164 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"60a4a4e0e49879528cdf55442dd62356","Type":"ContainerStarted","Data":"af1966b41412f5645ac46a26f5458e3dca82c3baaf8cc8887e1f02f97e1f9cd2"} Nov 04 00:57:41.281257 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.281189 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" event={"ID":"60a4a4e0e49879528cdf55442dd62356","Type":"ContainerStarted","Data":"05fc03a562394dad9dcc7c076bd724c757e3abc030e4aacc92b620e9fcb4de6e"} Nov 04 00:57:41.281257 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.281232 5051 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.281413 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.281349 5051 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.304574 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.304546 5051 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:41.304574 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.304555 5051 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:41.304768 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:41.304596 5051 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.304768 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:41.304598 5051 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.353062 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.352972 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60a4a4e0e49879528cdf55442dd62356-config\") pod \"kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"60a4a4e0e49879528cdf55442dd62356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.353062 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.353011 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.353062 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.353028 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.379109 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.379076 5051 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 00:57:41.382221 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.382204 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 04 00:57:41.382317 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.382238 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 04 00:57:41.382317 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.382251 5051 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-48.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 04 00:57:41.382317 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.382299 5051 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.393886 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.393865 5051 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.393989 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.393927 5051 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.454175 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.454081 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60a4a4e0e49879528cdf55442dd62356-config\") pod \"kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"60a4a4e0e49879528cdf55442dd62356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.454175 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.454162 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/60a4a4e0e49879528cdf55442dd62356-config\") pod \"kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"60a4a4e0e49879528cdf55442dd62356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.454402 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.454200 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.454402 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.454225 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.454402 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.454271 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:41.454402 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:41.454299 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a84db2d45c5cb12625222b3483c125f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" (UID: \"1a84db2d45c5cb12625222b3483c125f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:42.124881 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.124840 5051 apiserver.go:52] "Watching apiserver" Nov 04 00:57:42.134363 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.134335 5051 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 00:57:42.135045 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.135021 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tw9hp-q2fgs","openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth","openshift-image-registry/node-ca-dcksb","openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8","openshift-ingress-canary/ingress-canary-vl6w6","openshift-multus/multus-additional-cni-plugins-rlwsw","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn","openshift-network-diagnostics/network-check-target-zh5bh","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c","openshift-network-console/networking-console-plugin-86c478bb88-mh5d6","openshift-dns/node-resolver-shjsf","openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m","openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6","openshift-ingress/router-default-669749fcf7-vqftg","openshift-multus/multus-dm58x","kube-system/kas-connection-checker-685895c7c9-wmhkb","kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal","kube-system/kas-connection-checker-685895c7c9-bw7wx","openshift-ovn-kubernetes/ovnkube-node-j49c7","kube-system/global-pull-secret-syncer-8rwkl","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4","kube-system/kas-connection-checker-685895c7c9-hwnjk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal","openshift-dns/dns-default-mvtzk","openshift-image-registry/image-registry-75c85b76dc-nfpkl","openshift-multus/network-metrics-daemon-m84bl","openshift-insights/insights-runtime-extractor-wtpbl","kube-system/konnectivity-agent-w6tt8","openshift-image-registry/image-registry-6bfcf87bc8-zhslb","openshift-service-ca/service-ca-5fffc97bdf-2g6px","openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5","openshift-console-operator/console-operator-7cd56b8dbc-7vtl7","openshift-network-operator/iptables-alerter-fkvdp","openshift-cluster-node-tuning-operator/tuned-c4pg2","openshift-insights/insights-operator-7c4986d449-sgnvg"] Nov 04 00:57:42.136279 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.136254 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.137143 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.137112 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:42.137967 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.137948 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.139802 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.139784 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.139923 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.139906 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.140241 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.140226 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:42.140306 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.140255 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fjgn7\"" Nov 04 00:57:42.140306 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.140284 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.140427 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.140372 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mn9vp\"" Nov 04 00:57:42.140854 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.140819 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 00:57:42.141756 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.141735 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.141840 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.141801 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 00:57:42.142512 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.142490 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.143026 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143007 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 00:57:42.143110 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143089 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 00:57:42.143185 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143167 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.143428 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143410 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.143499 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143462 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9t5w2\"" Nov 04 00:57:42.143499 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143493 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 00:57:42.143640 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143625 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.143720 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143695 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 00:57:42.143823 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143807 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.143969 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.143954 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6kp4v\"" Nov 04 00:57:42.144257 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.144237 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 00:57:42.144331 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.144281 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 00:57:42.144419 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.144397 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.144534 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.144441 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 00:57:42.144591 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.144544 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 00:57:42.144591 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.144586 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.144817 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.144797 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7rnfb\"" Nov 04 00:57:42.144972 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.144953 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d6bjj\"" Nov 04 00:57:42.145041 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.145001 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rppzs\"" Nov 04 00:57:42.145134 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.145107 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 00:57:42.145286 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.145270 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.145521 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.145473 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.145691 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.145672 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 00:57:42.146675 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.146654 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.146758 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.146706 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.146863 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.146846 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.146916 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.146855 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 00:57:42.146964 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.146946 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.147008 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.146974 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 00:57:42.147065 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.147005 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 00:57:42.147321 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.147296 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 00:57:42.148862 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.148844 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 00:57:42.148955 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.148869 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mpjzr\"" Nov 04 00:57:42.148955 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.148915 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bcswz\"" Nov 04 00:57:42.149385 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149363 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.149472 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149399 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 00:57:42.149472 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149421 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 00:57:42.149472 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149441 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.149472 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149445 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-7dq87\"" Nov 04 00:57:42.149472 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149399 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.149472 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149458 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 00:57:42.149713 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149490 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.149888 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149872 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 00:57:42.149972 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149896 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 00:57:42.149972 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.149913 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.150213 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.150197 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 00:57:42.150268 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.150218 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 00:57:42.150305 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.150298 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gzs44\"" Nov 04 00:57:42.150464 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.150451 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:42.152047 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.152030 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.153211 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.153197 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.154086 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154065 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 00:57:42.154233 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154092 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sknxk\"" Nov 04 00:57:42.154233 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154103 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 00:57:42.154233 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154134 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 00:57:42.154438 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154403 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgrbh\"" Nov 04 00:57:42.154438 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154412 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.154438 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154409 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 00:57:42.154561 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154437 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.154561 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154420 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.154561 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154458 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.154561 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154489 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9jwd5\"" Nov 04 00:57:42.154561 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154496 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 00:57:42.154918 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154905 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.154970 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.154963 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.155070 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.155051 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xdt54\"" Nov 04 00:57:42.155433 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.155415 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 00:57:42.159986 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.159956 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.160092 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160074 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q48lg\"" Nov 04 00:57:42.160092 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160080 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.160281 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160263 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.160488 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160465 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 00:57:42.160542 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160521 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4rmxc\"" Nov 04 00:57:42.160703 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160688 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 00:57:42.160703 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160699 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-psr9r\"" Nov 04 00:57:42.160795 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160711 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 00:57:42.160795 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160777 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4k8lm\"" Nov 04 00:57:42.160886 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160825 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 00:57:42.160886 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160877 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.160972 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160903 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cjss9\"" Nov 04 00:57:42.160972 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160920 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 00:57:42.160972 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160965 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.161103 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160975 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 00:57:42.161103 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161017 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 00:57:42.161103 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161099 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.161257 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161136 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 00:57:42.161257 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.159957 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 00:57:42.161348 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161257 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.161348 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161276 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nszlw\"" Nov 04 00:57:42.161348 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161298 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-netd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.161348 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161339 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-netns\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161371 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-djh2q\"" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161393 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pb69q\"" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161400 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-multus-daemon-config\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.160902 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zx855\"" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161445 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161474 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kqwbk\"" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161484 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-config\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161510 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.161526 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161520 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3391f8ba-2fe6-4950-b644-5361fc8c2e48-host-slash\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161553 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-cnibin\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161560 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161584 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161624 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ln4qv\"" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161633 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-script-lib\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161662 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161677 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98tj\" (UniqueName: \"kubernetes.io/projected/d648b600-369c-47b0-ba02-6b3df1252bed-kube-api-access-n98tj\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161301 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161735 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.161880 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161794 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161902 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161983 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldjm6\"" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.161987 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-snapshots\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162053 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162135 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162174 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-snapshots\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162259 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-multus-daemon-config\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162272 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:42.162300 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162281 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 00:57:42.162761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162312 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-script-lib\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.162761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162348 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-config\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:42.162761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162349 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5r4\" (UniqueName: \"kubernetes.io/projected/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-kube-api-access-jv5r4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:42.162761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162461 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3391f8ba-2fe6-4950-b644-5361fc8c2e48-iptables-alerter-script\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:42.162761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162570 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29mnh\" (UniqueName: \"kubernetes.io/projected/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-kube-api-access-29mnh\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:42.162761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162617 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jddfl\" (UniqueName: \"kubernetes.io/projected/f4bef706-ca6f-4260-962f-ad0599ad093a-kube-api-access-jddfl\") pod \"kas-connection-checker-685895c7c9-bw7wx\" (UID: \"f4bef706-ca6f-4260-962f-ad0599ad093a\") " pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" Nov 04 00:57:42.162761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-service-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.163037 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162842 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 04 00:57:42.163037 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162886 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kwx\" (UniqueName: \"kubernetes.io/projected/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-kube-api-access-76kwx\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:42.163037 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162931 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-bin\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.163037 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162966 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-service-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.163037 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.162969 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-stats-auth\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.163267 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163038 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.163267 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163040 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-trusted-ca\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:42.163267 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163096 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-socket-dir-parent\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.163267 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163161 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aa9b05c-8730-4590-8709-0275e8be2631-serving-cert\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:42.163267 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163193 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aa9b05c-8730-4590-8709-0275e8be2631-config\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:42.163267 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163226 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-systemd-units\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.163267 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163257 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-node-log\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.163498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163268 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3391f8ba-2fe6-4950-b644-5361fc8c2e48-iptables-alerter-script\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:42.163498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163287 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-stats-auth\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.163498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163290 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.163498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163389 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-cnibin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.163498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163424 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aa9b05c-8730-4590-8709-0275e8be2631-config\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:42.163498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163431 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-etc-kubernetes\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.163498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163466 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.163498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163488 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhbb\" (UniqueName: \"kubernetes.io/projected/3391f8ba-2fe6-4950-b644-5361fc8c2e48-kube-api-access-kfhbb\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:42.163764 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163497 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aa9b05c-8730-4590-8709-0275e8be2631-serving-cert\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:42.163764 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163689 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-config\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.163764 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163737 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-env-overrides\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.163858 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163763 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-system-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.163858 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163799 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-k8s-cni-cncf-io\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.163858 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163825 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-multus\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.163858 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163853 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.163989 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163882 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskr9\" (UniqueName: \"kubernetes.io/projected/767984a4-597e-4498-a6bf-c988a7b228d1-kube-api-access-gskr9\") pod \"kas-connection-checker-685895c7c9-wmhkb\" (UID: \"767984a4-597e-4498-a6bf-c988a7b228d1\") " pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" Nov 04 00:57:42.163989 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163902 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.163989 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.163921 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbb6\" (UniqueName: \"kubernetes.io/projected/0c8ada62-5c11-4462-a29d-a576d7c162df-kube-api-access-mgbb6\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.164555 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.164526 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-var-lib-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.164694 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.164666 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.164831 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.164721 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-cni-binary-copy\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.164922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.164834 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d648b600-369c-47b0-ba02-6b3df1252bed-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.164922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.164878 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.164922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.164856 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovnkube-config\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.165092 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.164993 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.165092 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165004 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1006149a-92ee-43c1-bb41-6aaf1df41c36-env-overrides\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.165217 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165097 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-serving-cert\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:42.165217 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165105 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/08221c6a-1a73-4922-a769-caeb42c194bb-cni-binary-copy\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.165217 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165148 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.165217 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165189 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-kubelet\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.165217 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165187 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ede327f8-e1cf-4596-abe5-de379f528b15-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.165456 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165278 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d648b600-369c-47b0-ba02-6b3df1252bed-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.165456 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165312 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-serving-cert\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:42.165456 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165339 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-netns\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.165456 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165368 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:42.165456 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165394 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-hostroot\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.165456 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165416 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.165456 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165421 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165467 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165491 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165508 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-etc-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165523 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-kubelet\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165578 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165600 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b2w\" (UniqueName: \"kubernetes.io/projected/1006149a-92ee-43c1-bb41-6aaf1df41c36-kube-api-access-75b2w\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165620 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl6xd\" (UniqueName: \"kubernetes.io/projected/08221c6a-1a73-4922-a769-caeb42c194bb-kube-api-access-gl6xd\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165745 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.165782 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165755 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165787 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-system-cni-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165825 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165843 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovn-node-metrics-cert\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165871 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165892 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-default-certificate\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.165993 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-bin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166047 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1006149a-92ee-43c1-bb41-6aaf1df41c36-ovn-node-metrics-cert\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166090 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166098 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166149 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-ovn\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166176 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-default-certificate\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166188 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.166236 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166213 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-conf-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166274 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-multus-certs\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166318 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-tmp\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166370 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/261f5fa6-5956-4a5d-a104-982dc13aed39-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166427 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ede327f8-e1cf-4596-abe5-de379f528b15-tmp\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166429 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-binary-copy\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166455 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166480 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166483 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166519 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ede327f8-e1cf-4596-abe5-de379f528b15-serving-cert\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166544 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166564 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/261f5fa6-5956-4a5d-a104-982dc13aed39-nginx-conf\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:42.166631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166574 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc462\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166704 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72nkw\" (UniqueName: \"kubernetes.io/projected/7d024224-166d-4cb4-976f-75e0b47addae-kube-api-access-72nkw\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166738 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9x4t\" (UniqueName: \"kubernetes.io/projected/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd-kube-api-access-d9x4t\") pod \"volume-data-source-validator-58c9f5f46d-pv57c\" (UID: \"e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166754 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c8ada62-5c11-4462-a29d-a576d7c162df-cni-binary-copy\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166767 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rs4s\" (UniqueName: \"kubernetes.io/projected/969ca7d8-004d-4f22-9929-96099b5f011c-kube-api-access-5rs4s\") pod \"network-check-source-6f78d59f99-xqsk5\" (UID: \"969ca7d8-004d-4f22-9929-96099b5f011c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166822 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7l6w\" (UniqueName: \"kubernetes.io/projected/ac81ee2f-5c6a-4ead-89c5-2a181b928494-kube-api-access-x7l6w\") pod \"kas-connection-checker-685895c7c9-hwnjk\" (UID: \"ac81ee2f-5c6a-4ead-89c5-2a181b928494\") " pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166951 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzr6\" (UniqueName: \"kubernetes.io/projected/6aa9b05c-8730-4590-8709-0275e8be2631-kube-api-access-nwzr6\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.166984 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167018 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ede327f8-e1cf-4596-abe5-de379f528b15-serving-cert\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167021 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.167061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167053 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-slash\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.167478 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167081 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlnpx\" (UniqueName: \"kubernetes.io/projected/ede327f8-e1cf-4596-abe5-de379f528b15-kube-api-access-xlnpx\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.167478 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167099 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.167478 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167141 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-systemd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.167478 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167218 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-log-socket\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.167478 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167274 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:42.167478 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.167348 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:42.168211 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.168191 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 04 00:57:42.168726 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.168711 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 04 00:57:42.169385 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.169371 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 04 00:57:42.169643 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.169629 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-52wlr\"" Nov 04 00:57:42.171483 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.171467 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98tj\" (UniqueName: \"kubernetes.io/projected/d648b600-369c-47b0-ba02-6b3df1252bed-kube-api-access-n98tj\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.171605 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.171588 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jddfl\" (UniqueName: \"kubernetes.io/projected/f4bef706-ca6f-4260-962f-ad0599ad093a-kube-api-access-jddfl\") pod \"kas-connection-checker-685895c7c9-bw7wx\" (UID: \"f4bef706-ca6f-4260-962f-ad0599ad093a\") " pod="kube-system/kas-connection-checker-685895c7c9-bw7wx" Nov 04 00:57:42.172828 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.172804 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbb6\" (UniqueName: \"kubernetes.io/projected/0c8ada62-5c11-4462-a29d-a576d7c162df-kube-api-access-mgbb6\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.173205 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.173189 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhbb\" (UniqueName: \"kubernetes.io/projected/3391f8ba-2fe6-4950-b644-5361fc8c2e48-kube-api-access-kfhbb\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:42.173593 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.173565 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskr9\" (UniqueName: \"kubernetes.io/projected/767984a4-597e-4498-a6bf-c988a7b228d1-kube-api-access-gskr9\") pod \"kas-connection-checker-685895c7c9-wmhkb\" (UID: \"767984a4-597e-4498-a6bf-c988a7b228d1\") " pod="kube-system/kas-connection-checker-685895c7c9-wmhkb" Nov 04 00:57:42.173738 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.173608 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rs4s\" (UniqueName: \"kubernetes.io/projected/969ca7d8-004d-4f22-9929-96099b5f011c-kube-api-access-5rs4s\") pod \"network-check-source-6f78d59f99-xqsk5\" (UID: \"969ca7d8-004d-4f22-9929-96099b5f011c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-xqsk5" Nov 04 00:57:42.173906 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.173886 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl6xd\" (UniqueName: \"kubernetes.io/projected/08221c6a-1a73-4922-a769-caeb42c194bb-kube-api-access-gl6xd\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.174010 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.173992 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b2w\" (UniqueName: \"kubernetes.io/projected/1006149a-92ee-43c1-bb41-6aaf1df41c36-kube-api-access-75b2w\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.174061 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.174042 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kwx\" (UniqueName: \"kubernetes.io/projected/9747f7d7-39f9-452d-9e9a-dbc9de48c89e-kube-api-access-76kwx\") pod \"console-operator-7cd56b8dbc-7vtl7\" (UID: \"9747f7d7-39f9-452d-9e9a-dbc9de48c89e\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:42.174096 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.174078 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5r4\" (UniqueName: \"kubernetes.io/projected/2f2b307f-e5f0-4116-b2cd-2bce47ae5b86-kube-api-access-jv5r4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4\" (UID: \"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" Nov 04 00:57:42.174842 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.174827 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29mnh\" (UniqueName: \"kubernetes.io/projected/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-kube-api-access-29mnh\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:42.175647 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.175624 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72nkw\" (UniqueName: \"kubernetes.io/projected/7d024224-166d-4cb4-976f-75e0b47addae-kube-api-access-72nkw\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.176686 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.176671 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.176978 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.176961 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlnpx\" (UniqueName: \"kubernetes.io/projected/ede327f8-e1cf-4596-abe5-de379f528b15-kube-api-access-xlnpx\") pod \"insights-operator-7c4986d449-sgnvg\" (UID: \"ede327f8-e1cf-4596-abe5-de379f528b15\") " pod="openshift-insights/insights-operator-7c4986d449-sgnvg" Nov 04 00:57:42.180392 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.180370 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzr6\" (UniqueName: \"kubernetes.io/projected/6aa9b05c-8730-4590-8709-0275e8be2631-kube-api-access-nwzr6\") pod \"service-ca-operator-544d749c8c-22cr8\" (UID: \"6aa9b05c-8730-4590-8709-0275e8be2631\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" Nov 04 00:57:42.180992 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.180970 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9x4t\" (UniqueName: \"kubernetes.io/projected/e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd-kube-api-access-d9x4t\") pod \"volume-data-source-validator-58c9f5f46d-pv57c\" (UID: \"e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pv57c" Nov 04 00:57:42.181180 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.181161 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc462\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.181543 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.181527 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7l6w\" (UniqueName: \"kubernetes.io/projected/ac81ee2f-5c6a-4ead-89c5-2a181b928494-kube-api-access-x7l6w\") pod \"kas-connection-checker-685895c7c9-hwnjk\" (UID: \"ac81ee2f-5c6a-4ead-89c5-2a181b928494\") " pod="kube-system/kas-connection-checker-685895c7c9-hwnjk" Nov 04 00:57:42.243821 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.243792 5051 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:42.243987 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.243876 5051 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:42.244978 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.244961 5051 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 00:57:42.257614 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.257597 5051 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:42.257614 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.257597 5051 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 00:57:42.257728 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.257652 5051 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:42.257728 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.257698 5051 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-48.us-east-2.compute.internal" Nov 04 00:57:42.267966 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.267939 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.267966 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.267967 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-ovn\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.268145 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.267988 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-modprobe-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268145 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268005 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysconfig\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268145 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268051 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268145 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268078 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-conf\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268145 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268110 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-ovn\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.268314 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268138 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-dbus\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:42.268314 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268173 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.268314 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268198 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-host\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268314 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268215 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.268314 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268231 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:42.268482 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268331 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-tmp\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268482 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268358 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sb2n\" (UniqueName: \"kubernetes.io/projected/51fbfff7-c90b-4726-bbff-6ebbcc943626-kube-api-access-6sb2n\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268482 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268383 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbd7a56-ab21-413c-abac-36b1f79f452e-cert\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:42.268482 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.268415 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.768395159 +0000 UTC m=+1.740144022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:42.268482 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268433 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-var-lib-kubelet\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268482 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268458 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-kubelet-config\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268484 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-tmp\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268526 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268547 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268587 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-slash\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268624 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268649 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-log-socket\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268686 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-slash\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268699 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-log-socket\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268740 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-netns\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.268761 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.268731 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268764 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbd7a56-ab21-413c-abac-36b1f79f452e-cert\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268769 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ff9909-e56c-4d27-be31-574bf33a8f75-tmp-dir\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.268769 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c85b76dc-nfpkl: secret "image-registry-tls" not found Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268806 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-netns\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268819 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-cabundle\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268821 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268831 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.268856 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls podName:3603f9b4-53ae-4e45-832f-e5c29ca743b9 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.768844253 +0000 UTC m=+1.740593102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls") pod "image-registry-75c85b76dc-nfpkl" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9") : secret "image-registry-tls" not found Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268874 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ff9909-e56c-4d27-be31-574bf33a8f75-tmp-dir\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.268883 5051 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268907 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3391f8ba-2fe6-4950-b644-5361fc8c2e48-host-slash\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268873 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3391f8ba-2fe6-4950-b644-5361fc8c2e48-host-slash\") pod \"iptables-alerter-fkvdp\" (UID: \"3391f8ba-2fe6-4950-b644-5361fc8c2e48\") " pod="openshift-network-operator/iptables-alerter-fkvdp" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.268942 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert podName:261f5fa6-5956-4a5d-a104-982dc13aed39 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.768930936 +0000 UTC m=+1.740679810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-mh5d6" (UID: "261f5fa6-5956-4a5d-a104-982dc13aed39") : secret "networking-console-plugin-cert" not found Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268970 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-cnibin\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.268998 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269018 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlml8\" (UniqueName: \"kubernetes.io/projected/5b2c9231-5813-4143-93ca-2457278f37a2-kube-api-access-nlml8\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269039 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-cnibin\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269035 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-agent-certs\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:42.269115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269089 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjjp\" (UniqueName: \"kubernetes.io/projected/51ff9909-e56c-4d27-be31-574bf33a8f75-kube-api-access-9cjjp\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269137 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-sys-fs\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269192 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-bin\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269225 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269250 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-socket-dir-parent\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269276 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxj6\" (UniqueName: \"kubernetes.io/projected/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-kube-api-access-htxj6\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269286 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-bin\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269305 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-agent-certs\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269308 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-registration-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269329 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269337 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-socket-dir-parent\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269373 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-node-log\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269416 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269424 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-node-log\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269464 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-cnibin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269554 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-etc-kubernetes\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269580 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-cnibin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269591 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8d5k\" (UniqueName: \"kubernetes.io/projected/756d0c06-26e9-4c88-8520-34f21fa6a9bd-kube-api-access-w8d5k\") pod \"migrator-66df5d496b-n84lh\" (UID: \"756d0c06-26e9-4c88-8520-34f21fa6a9bd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269672 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-etc-kubernetes\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269707 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-k8s-cni-cncf-io\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269723 5051 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269740 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-multus\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269765 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-k8s-cni-cncf-io\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269770 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95cq\" (UniqueName: \"kubernetes.io/projected/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kube-api-access-m95cq\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269811 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-var-lib-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269840 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269866 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-data-volume\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269890 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk25v\" (UniqueName: \"kubernetes.io/projected/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-api-access-hk25v\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269908 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-var-lib-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269917 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51ff9909-e56c-4d27-be31-574bf33a8f75-hosts-file\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269945 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-kubernetes\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.269957 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269964 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-multus\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269984 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.269999 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-data-volume\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270073 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-device-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270137 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7xnz\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270257 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270315 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270338 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-etc-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270354 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-run\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270374 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270407 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dqxb\" (UniqueName: \"kubernetes.io/projected/b0807c2d-b756-42e8-86b3-97c30f23df72-kube-api-access-5dqxb\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270405 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270424 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270458 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-conf-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270477 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270481 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-multus-certs\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270512 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-run-multus-certs\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270522 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-key\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270543 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-etc-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270554 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270588 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-socket-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270617 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270644 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0807c2d-b756-42e8-86b3-97c30f23df72-config-volume\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270673 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-crio-socket\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270684 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270707 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-systemd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270733 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-netd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270759 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270765 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-signing-key\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270786 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270835 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-openvswitch\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270895 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270951 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-multus-conf-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.270979 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-cni-netd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271017 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-run-systemd\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271020 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0807c2d-b756-42e8-86b3-97c30f23df72-config-volume\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271053 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271099 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271100 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271171 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-tuned\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.271178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271199 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-serviceca\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271225 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271256 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swhr9\" (UniqueName: \"kubernetes.io/projected/903f3694-f0a7-4d5f-828b-20bced1b1066-kube-api-access-swhr9\") pod \"migrator-66df5d496b-lbr5m\" (UID: \"903f3694-f0a7-4d5f-828b-20bced1b1066\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271346 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnff8\" (UniqueName: \"kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271358 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-tuned\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271480 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271507 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271589 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271709 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-systemd-units\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271750 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-sys\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271777 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271787 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-systemd-units\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271820 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-system-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271845 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-systemd\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271875 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-host\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271906 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271953 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04f82b22-3949-44d3-abe9-6473a2f68f76-original-pull-secret\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.271955 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-kubelet\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272008 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-kubelet\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272047 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-netns\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272084 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272105 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1006149a-92ee-43c1-bb41-6aaf1df41c36-host-run-netns\") pod \"ovnkube-node-j49c7\" (UID: \"1006149a-92ee-43c1-bb41-6aaf1df41c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272116 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-hostroot\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272158 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272181 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-system-cni-dir\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272216 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-konnectivity-ca\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272245 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-kubelet\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.272251 5051 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272289 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-serviceca\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272290 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272340 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0807c2d-b756-42e8-86b3-97c30f23df72-tmp-dir\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272343 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272414 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0807c2d-b756-42e8-86b3-97c30f23df72-tmp-dir\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.272417 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls podName:8cf9dde4-e870-49c8-a34d-1c0a4c85acb6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.772401958 +0000 UTC m=+1.744150825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69rhn" (UID: "8cf9dde4-e870-49c8-a34d-1c0a4c85acb6") : secret "samples-operator-tls" not found Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272412 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ef08de9-97a2-4c5b-ae3a-66a9cd783a31-konnectivity-ca\") pod \"konnectivity-agent-w6tt8\" (UID: \"6ef08de9-97a2-4c5b-ae3a-66a9cd783a31\") " pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272451 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272466 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-hostroot\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272500 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-kubelet\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272500 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-lib-modules\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272528 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6b48\" (UniqueName: \"kubernetes.io/projected/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-kube-api-access-k6b48\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272565 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvpsh\" (UniqueName: \"kubernetes.io/projected/8dbd7a56-ab21-413c-abac-36b1f79f452e-kube-api-access-tvpsh\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272589 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-system-cni-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272615 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-bin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272570 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d648b600-369c-47b0-ba02-6b3df1252bed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-kxzp6\" (UID: \"d648b600-369c-47b0-ba02-6b3df1252bed\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272627 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272707 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c8ada62-5c11-4462-a29d-a576d7c162df-system-cni-dir\") pod \"multus-additional-cni-plugins-rlwsw\" (UID: \"0c8ada62-5c11-4462-a29d-a576d7c162df\") " pod="openshift-multus/multus-additional-cni-plugins-rlwsw" Nov 04 00:57:42.272939 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.272730 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08221c6a-1a73-4922-a769-caeb42c194bb-host-var-lib-cni-bin\") pod \"multus-dm58x\" (UID: \"08221c6a-1a73-4922-a769-caeb42c194bb\") " pod="openshift-multus/multus-dm58x" Nov 04 00:57:42.274724 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.274704 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d024224-166d-4cb4-976f-75e0b47addae-metrics-certs\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.276041 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.276024 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sb2n\" (UniqueName: \"kubernetes.io/projected/51fbfff7-c90b-4726-bbff-6ebbcc943626-kube-api-access-6sb2n\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.276861 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.276840 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlml8\" (UniqueName: \"kubernetes.io/projected/5b2c9231-5813-4143-93ca-2457278f37a2-kube-api-access-nlml8\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:42.277489 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.277466 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnff8\" (UniqueName: \"kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8\") pod \"periodic-gathering-tw9hp-q2fgs\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:57:42.277661 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.277643 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95cq\" (UniqueName: \"kubernetes.io/projected/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kube-api-access-m95cq\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.278747 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.278731 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk25v\" (UniqueName: \"kubernetes.io/projected/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-api-access-hk25v\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.279438 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.279422 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8d5k\" (UniqueName: \"kubernetes.io/projected/756d0c06-26e9-4c88-8520-34f21fa6a9bd-kube-api-access-w8d5k\") pod \"migrator-66df5d496b-n84lh\" (UID: \"756d0c06-26e9-4c88-8520-34f21fa6a9bd\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n84lh" Nov 04 00:57:42.280498 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.280474 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjjp\" (UniqueName: \"kubernetes.io/projected/51ff9909-e56c-4d27-be31-574bf33a8f75-kube-api-access-9cjjp\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:42.280589 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.280519 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxj6\" (UniqueName: \"kubernetes.io/projected/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-kube-api-access-htxj6\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:42.280589 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.280535 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swhr9\" (UniqueName: \"kubernetes.io/projected/903f3694-f0a7-4d5f-828b-20bced1b1066-kube-api-access-swhr9\") pod \"migrator-66df5d496b-lbr5m\" (UID: \"903f3694-f0a7-4d5f-828b-20bced1b1066\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lbr5m" Nov 04 00:57:42.280671 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.280657 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswcp\" (UniqueName: \"kubernetes.io/projected/66c16477-f3db-43cf-9830-39fe0267bdce-kube-api-access-xswcp\") pod \"network-check-target-zh5bh\" (UID: \"66c16477-f3db-43cf-9830-39fe0267bdce\") " pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:42.281320 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.281300 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dqxb\" (UniqueName: \"kubernetes.io/projected/b0807c2d-b756-42e8-86b3-97c30f23df72-kube-api-access-5dqxb\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.281429 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.281412 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7xnz\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.285296 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.285277 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.285812 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.285781 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvpsh\" (UniqueName: \"kubernetes.io/projected/8dbd7a56-ab21-413c-abac-36b1f79f452e-kube-api-access-tvpsh\") pod \"ingress-canary-vl6w6\" (UID: \"8dbd7a56-ab21-413c-abac-36b1f79f452e\") " pod="openshift-ingress-canary/ingress-canary-vl6w6" Nov 04 00:57:42.286031 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.286015 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6b48\" (UniqueName: \"kubernetes.io/projected/7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac-kube-api-access-k6b48\") pod \"service-ca-5fffc97bdf-2g6px\" (UID: \"7c3b4dfd-5d4c-4ac6-b46f-17a9755f03ac\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2g6px" Nov 04 00:57:42.373660 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373626 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-socket-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.373660 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373664 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-crio-socket\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373683 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373698 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373717 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373734 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373758 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-sys\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373775 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-crio-socket\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373801 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373823 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-sys\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373782 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-socket-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373830 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.373843 5051 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 00:57:42.373914 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373877 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-systemd\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373968 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-host\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373980 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-systemd\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.373994 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b7cc6df-efb4-4ed5-a299-99d3eafd8e56-host\") pod \"node-ca-dcksb\" (UID: \"5b7cc6df-efb4-4ed5-a299-99d3eafd8e56\") " pod="openshift-image-registry/node-ca-dcksb" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.374012 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.873990225 +0000 UTC m=+1.845739086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : secret "metrics-daemon-secret" not found Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374059 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-lib-modules\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374080 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-modprobe-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374094 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysconfig\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374108 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374151 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-conf\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374180 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-dbus\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374187 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysconfig\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374232 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-host\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374235 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-modprobe-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374249 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-dbus\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374246 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-lib-modules\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374261 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374281 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-conf\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374286 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-host\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374298 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-var-lib-kubelet\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374331 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-kubelet-config\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374339 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-var-lib-kubelet\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.374354 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.374366 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bfcf87bc8-zhslb: secret "image-registry-tls" not found Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374385 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-sysctl-d\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.374379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374393 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374386 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04f82b22-3949-44d3-abe9-6473a2f68f76-kubelet-config\") pod \"global-pull-secret-syncer-8rwkl\" (UID: \"04f82b22-3949-44d3-abe9-6473a2f68f76\") " pod="kube-system/global-pull-secret-syncer-8rwkl" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.374413 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls podName:e54204af-fcd5-442e-9009-f58ba803f3c4 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.874398022 +0000 UTC m=+1.846146888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls") pod "image-registry-6bfcf87bc8-zhslb" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4") : secret "image-registry-tls" not found Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374444 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-sys-fs\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.374453 5051 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-registration-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374500 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51ff9909-e56c-4d27-be31-574bf33a8f75-hosts-file\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374501 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-sys-fs\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374515 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-kubernetes\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374537 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-registration-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.374548 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:42.874524397 +0000 UTC m=+1.846273246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374585 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374620 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-device-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374654 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-run\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374552 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51ff9909-e56c-4d27-be31-574bf33a8f75-hosts-file\") pod \"node-resolver-shjsf\" (UID: \"51ff9909-e56c-4d27-be31-574bf33a8f75\") " pod="openshift-dns/node-resolver-shjsf" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374556 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-etc-kubernetes\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374694 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7-device-dir\") pod \"aws-ebs-csi-driver-node-6jrth\" (UID: \"47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.374764 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51fbfff7-c90b-4726-bbff-6ebbcc943626-run\") pod \"tuned-c4pg2\" (UID: \"51fbfff7-c90b-4726-bbff-6ebbcc943626\") " pod="openshift-cluster-node-tuning-operator/tuned-c4pg2" Nov 04 00:57:42.375411 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.375169 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.376212 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.376196 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0807c2d-b756-42e8-86b3-97c30f23df72-metrics-tls\") pod \"dns-default-mvtzk\" (UID: \"b0807c2d-b756-42e8-86b3-97c30f23df72\") " pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.437062 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.437027 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" Nov 04 00:57:42.438982 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.438950 5051 scope.go:122] "RemoveContainer" containerID="8b1ac2c33f7a81e7550c8cc5e675d9803e8f2307b5e6c85fbbd6e1a0817c6094" Nov 04 00:57:42.454238 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.454215 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:42.586073 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.586041 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:42.717633 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.717563 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6"] Nov 04 00:57:42.720772 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:42.720734 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd648b600_369c_47b0_ba02_6b3df1252bed.slice/crio-e092c1a07cfcbae52c43657a032ddf88e0786ff9d5ceeace9e4c1c7df6b66012 WatchSource:0}: Error finding container e092c1a07cfcbae52c43657a032ddf88e0786ff9d5ceeace9e4c1c7df6b66012: Status 404 returned error can't find the container with id e092c1a07cfcbae52c43657a032ddf88e0786ff9d5ceeace9e4c1c7df6b66012 Nov 04 00:57:42.723509 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.723490 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 00:57:42.743954 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.743924 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mvtzk"] Nov 04 00:57:42.747331 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:42.747306 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0807c2d_b756_42e8_86b3_97c30f23df72.slice/crio-3f7a08191c0f222cd67d82d75ca6c060a26f0d8f0b3c4fbd6ccf42ed375fe049 WatchSource:0}: Error finding container 3f7a08191c0f222cd67d82d75ca6c060a26f0d8f0b3c4fbd6ccf42ed375fe049: Status 404 returned error can't find the container with id 3f7a08191c0f222cd67d82d75ca6c060a26f0d8f0b3c4fbd6ccf42ed375fe049 Nov 04 00:57:42.777329 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.777300 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:42.777473 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.777344 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:42.777473 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.777367 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:42.777473 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.777433 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:42.777473 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.777463 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:42.777615 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.777478 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:43.777457989 +0000 UTC m=+2.749206838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:42.777615 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.777484 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c85b76dc-nfpkl: secret "image-registry-tls" not found Nov 04 00:57:42.777615 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.777518 5051 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 00:57:42.777615 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.777528 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls podName:3603f9b4-53ae-4e45-832f-e5c29ca743b9 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:43.777517329 +0000 UTC m=+2.749266196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls") pod "image-registry-75c85b76dc-nfpkl" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9") : secret "image-registry-tls" not found Nov 04 00:57:42.777615 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.777554 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls podName:8cf9dde4-e870-49c8-a34d-1c0a4c85acb6 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:43.777543961 +0000 UTC m=+2.749292835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-69rhn" (UID: "8cf9dde4-e870-49c8-a34d-1c0a4c85acb6") : secret "samples-operator-tls" not found Nov 04 00:57:42.779777 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.779754 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/261f5fa6-5956-4a5d-a104-982dc13aed39-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-mh5d6\" (UID: \"261f5fa6-5956-4a5d-a104-982dc13aed39\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:42.880467 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.880428 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:42.880638 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.880513 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:42.880638 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.880564 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:42.880638 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.880588 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bfcf87bc8-zhslb: secret "image-registry-tls" not found Nov 04 00:57:42.880638 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:42.880598 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:42.880847 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.880634 5051 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:42.880847 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.880654 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls podName:e54204af-fcd5-442e-9009-f58ba803f3c4 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:43.880634994 +0000 UTC m=+2.852383863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls") pod "image-registry-6bfcf87bc8-zhslb" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4") : secret "image-registry-tls" not found Nov 04 00:57:42.880847 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.880701 5051 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 00:57:42.880847 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.880710 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:43.880693165 +0000 UTC m=+2.852442030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:42.880847 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:42.880744 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:43.880731091 +0000 UTC m=+2.852479947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : secret "metrics-daemon-secret" not found Nov 04 00:57:43.041477 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.041439 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" Nov 04 00:57:43.248249 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.248213 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvtzk" event={"ID":"b0807c2d-b756-42e8-86b3-97c30f23df72","Type":"ContainerStarted","Data":"3f7a08191c0f222cd67d82d75ca6c060a26f0d8f0b3c4fbd6ccf42ed375fe049"} Nov 04 00:57:43.249399 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.249379 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" event={"ID":"d648b600-369c-47b0-ba02-6b3df1252bed","Type":"ContainerStarted","Data":"e092c1a07cfcbae52c43657a032ddf88e0786ff9d5ceeace9e4c1c7df6b66012"} Nov 04 00:57:43.251153 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.251110 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 00:57:43.251285 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.251206 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" event={"ID":"9747f7d7-39f9-452d-9e9a-dbc9de48c89e","Type":"ContainerStarted","Data":"e66f6825950ca2e9bca4cbff589374db0f2b585f5f03383ede1ad8c4ef78f232"} Nov 04 00:57:43.314738 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.314663 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-mh5d6"] Nov 04 00:57:43.318786 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:43.318757 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261f5fa6_5956_4a5d_a104_982dc13aed39.slice/crio-e1edc444475edc47d3fba3f7af9901db3e610a5a9edd658c0a25d7bc6e2a5919 WatchSource:0}: Error finding container e1edc444475edc47d3fba3f7af9901db3e610a5a9edd658c0a25d7bc6e2a5919: Status 404 returned error can't find the container with id e1edc444475edc47d3fba3f7af9901db3e610a5a9edd658c0a25d7bc6e2a5919 Nov 04 00:57:43.789820 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.789776 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:43.789976 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.789851 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:43.789976 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.789949 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:43.790305 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.790279 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:43.790414 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.790308 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75c85b76dc-nfpkl: secret "image-registry-tls" not found Nov 04 00:57:43.790414 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.790296 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:45.790271783 +0000 UTC m=+4.762020649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:43.790414 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.790382 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls podName:3603f9b4-53ae-4e45-832f-e5c29ca743b9 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:45.790363506 +0000 UTC m=+4.762112355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls") pod "image-registry-75c85b76dc-nfpkl" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9") : secret "image-registry-tls" not found Nov 04 00:57:43.793011 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.792982 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf9dde4-e870-49c8-a34d-1c0a4c85acb6-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-69rhn\" (UID: \"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:43.891600 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.891430 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:43.891600 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.891525 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:43.891600 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.891559 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:43.891848 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.891717 5051 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:43.891848 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.891779 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:45.891765821 +0000 UTC m=+4.863514670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:43.891848 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.891833 5051 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 00:57:43.892206 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.891855 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs podName:5b2c9231-5813-4143-93ca-2457278f37a2 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:45.891847974 +0000 UTC m=+4.863596822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs") pod "network-metrics-daemon-m84bl" (UID: "5b2c9231-5813-4143-93ca-2457278f37a2") : secret "metrics-daemon-secret" not found Nov 04 00:57:43.892206 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.891899 5051 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 00:57:43.892206 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.891911 5051 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bfcf87bc8-zhslb: secret "image-registry-tls" not found Nov 04 00:57:43.892206 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:43.891943 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls podName:e54204af-fcd5-442e-9009-f58ba803f3c4 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:45.891933724 +0000 UTC m=+4.863682576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls") pod "image-registry-6bfcf87bc8-zhslb" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4") : secret "image-registry-tls" not found Nov 04 00:57:43.938054 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:43.937800 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" Nov 04 00:57:44.225675 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:44.225502 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn"] Nov 04 00:57:44.258442 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:44.258386 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" event={"ID":"261f5fa6-5956-4a5d-a104-982dc13aed39","Type":"ContainerStarted","Data":"e1edc444475edc47d3fba3f7af9901db3e610a5a9edd658c0a25d7bc6e2a5919"} Nov 04 00:57:45.264650 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.264598 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" event={"ID":"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6","Type":"ContainerStarted","Data":"85943fb56cf7dd4e7611a9c5a77f178355224982916a525c3bca2971b2fe0028"} Nov 04 00:57:45.266644 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.266615 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-kxzp6" event={"ID":"d648b600-369c-47b0-ba02-6b3df1252bed","Type":"ContainerStarted","Data":"16c72a75779d366ea4ec27f4fbfa77554593c09826bbf06912523b2bdf1ed5c6"} Nov 04 00:57:45.268428 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.268403 5051 generic.go:356] "Generic (PLEG): container finished" podID="2f2b307f-e5f0-4116-b2cd-2bce47ae5b86" containerID="722e517c22708269134694f3f5d833ed958ae5d513927fd9c6d3bdab9056f6a3" exitCode=0 Nov 04 00:57:45.268593 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.268480 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" event={"ID":"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86","Type":"ContainerDied","Data":"722e517c22708269134694f3f5d833ed958ae5d513927fd9c6d3bdab9056f6a3"} Nov 04 00:57:45.268833 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.268814 5051 scope.go:122] "RemoveContainer" containerID="722e517c22708269134694f3f5d833ed958ae5d513927fd9c6d3bdab9056f6a3" Nov 04 00:57:45.270793 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.270768 5051 generic.go:356] "Generic (PLEG): container finished" podID="ede327f8-e1cf-4596-abe5-de379f528b15" containerID="00e74c9e1ed136bcdb85b5aef14228d27ea9e91ab2058c6cf4dc6df1668155c1" exitCode=0 Nov 04 00:57:45.270897 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.270804 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" event={"ID":"ede327f8-e1cf-4596-abe5-de379f528b15","Type":"ContainerDied","Data":"00e74c9e1ed136bcdb85b5aef14228d27ea9e91ab2058c6cf4dc6df1668155c1"} Nov 04 00:57:45.271115 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.271099 5051 scope.go:122] "RemoveContainer" containerID="00e74c9e1ed136bcdb85b5aef14228d27ea9e91ab2058c6cf4dc6df1668155c1" Nov 04 00:57:45.273094 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.273073 5051 generic.go:356] "Generic (PLEG): container finished" podID="6aa9b05c-8730-4590-8709-0275e8be2631" containerID="c328e5385563653eddd50c01ddd929a6663093428d488cf01b874d042c1ab541" exitCode=0 Nov 04 00:57:45.273187 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.273165 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" event={"ID":"6aa9b05c-8730-4590-8709-0275e8be2631","Type":"ContainerDied","Data":"c328e5385563653eddd50c01ddd929a6663093428d488cf01b874d042c1ab541"} Nov 04 00:57:45.273467 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.273450 5051 scope.go:122] "RemoveContainer" containerID="c328e5385563653eddd50c01ddd929a6663093428d488cf01b874d042c1ab541" Nov 04 00:57:45.279205 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.279180 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvtzk" event={"ID":"b0807c2d-b756-42e8-86b3-97c30f23df72","Type":"ContainerStarted","Data":"6d4aa12ba885621bc74e74e499accfad5db7ff503503dee1c9143ec92acaadb0"} Nov 04 00:57:45.279306 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.279230 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvtzk" event={"ID":"b0807c2d-b756-42e8-86b3-97c30f23df72","Type":"ContainerStarted","Data":"1b152465cb1309f5265d84cab8c47a9dde6f34e9eb0b7dd40893229c8173aba1"} Nov 04 00:57:45.279573 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.279525 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:45.281446 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.281423 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-mh5d6" event={"ID":"261f5fa6-5956-4a5d-a104-982dc13aed39","Type":"ContainerStarted","Data":"0aa1d91edfec78167d0b398686d83492b62e6dc266706558063d8b016c890978"} Nov 04 00:57:45.516145 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.515465 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:57:45.819570 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.818596 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:45.819570 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.818663 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:45.819570 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:45.819246 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle podName:7d024224-166d-4cb4-976f-75e0b47addae nodeName:}" failed. No retries permitted until 2026-11-04 00:57:49.819226884 +0000 UTC m=+8.790975733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle") pod "router-default-669749fcf7-vqftg" (UID: "7d024224-166d-4cb4-976f-75e0b47addae") : configmap references non-existent config key: service-ca.crt Nov 04 00:57:45.824111 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.824083 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"image-registry-75c85b76dc-nfpkl\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:45.920317 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.920214 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:45.920500 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.920427 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:45.920500 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.920478 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:45.920646 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:45.920616 5051 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 00:57:45.920697 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:57:45.920692 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls podName:e7f2ccca-45df-41f8-bbf9-0b8aaa791160 nodeName:}" failed. No retries permitted until 2026-11-04 00:57:49.920673467 +0000 UTC m=+8.892422325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wtpbl" (UID: "e7f2ccca-45df-41f8-bbf9-0b8aaa791160") : secret "insights-runtime-extractor-tls" not found Nov 04 00:57:45.923243 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.923214 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b2c9231-5813-4143-93ca-2457278f37a2-metrics-certs\") pod \"network-metrics-daemon-m84bl\" (UID: \"5b2c9231-5813-4143-93ca-2457278f37a2\") " pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:45.924626 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:45.924606 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"image-registry-6bfcf87bc8-zhslb\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:46.042688 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.042650 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:46.050943 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.050910 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m84bl" Nov 04 00:57:46.053106 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.053076 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:46.288026 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.287969 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" event={"ID":"ede327f8-e1cf-4596-abe5-de379f528b15","Type":"ContainerStarted","Data":"307095b96a3aaff66e734627f934e11ddc0fc80f5b6d40da63ea7ae5be5fd243"} Nov 04 00:57:46.290155 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.290128 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-22cr8" event={"ID":"6aa9b05c-8730-4590-8709-0275e8be2631","Type":"ContainerStarted","Data":"1e71f42bed48b1a1b03f1f36bd0030f867a740d1377e326008111e76ff3f543a"} Nov 04 00:57:46.292372 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.292343 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n4jk4" event={"ID":"2f2b307f-e5f0-4116-b2cd-2bce47ae5b86","Type":"ContainerStarted","Data":"8696b1384732aed46cefd29b7fb46d08933d41ba97f82bb2a2b13cd7318315ce"} Nov 04 00:57:46.346213 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.346186 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m84bl"] Nov 04 00:57:46.346319 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.346260 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bfcf87bc8-zhslb"] Nov 04 00:57:46.348355 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:46.348328 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2c9231_5813_4143_93ca_2457278f37a2.slice/crio-4a111913a64983d56c0d747fdb74e1fe26dd0f7f17627a612b24ce7e5b6202e4 WatchSource:0}: Error finding container 4a111913a64983d56c0d747fdb74e1fe26dd0f7f17627a612b24ce7e5b6202e4: Status 404 returned error can't find the container with id 4a111913a64983d56c0d747fdb74e1fe26dd0f7f17627a612b24ce7e5b6202e4 Nov 04 00:57:46.350640 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:46.350615 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54204af_fcd5_442e_9009_f58ba803f3c4.slice/crio-3679af22f4dd6a21469926a4b18d7c59851fca3d2edb01992667477f6f4c88d9 WatchSource:0}: Error finding container 3679af22f4dd6a21469926a4b18d7c59851fca3d2edb01992667477f6f4c88d9: Status 404 returned error can't find the container with id 3679af22f4dd6a21469926a4b18d7c59851fca3d2edb01992667477f6f4c88d9 Nov 04 00:57:46.352942 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:46.352874 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75c85b76dc-nfpkl"] Nov 04 00:57:46.365350 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:46.365274 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3603f9b4_53ae_4e45_832f_e5c29ca743b9.slice/crio-f947618ec4af545ba4640a197fa4a4ba8bcb571be9e5f7c09d91a26665a88d6e WatchSource:0}: Error finding container f947618ec4af545ba4640a197fa4a4ba8bcb571be9e5f7c09d91a26665a88d6e: Status 404 returned error can't find the container with id f947618ec4af545ba4640a197fa4a4ba8bcb571be9e5f7c09d91a26665a88d6e Nov 04 00:57:47.302012 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:47.301971 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" event={"ID":"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6","Type":"ContainerStarted","Data":"7844771c7d283f9a1ef8b8ec372d1f607440ed10566cf545816c497f9d2a4965"} Nov 04 00:57:47.302518 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:47.302070 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-69rhn" event={"ID":"8cf9dde4-e870-49c8-a34d-1c0a4c85acb6","Type":"ContainerStarted","Data":"3bd0a3efb178953c7d024129ab4395594fe8ea7716067301de5f58bb00676552"} Nov 04 00:57:47.303748 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:47.303700 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" event={"ID":"3603f9b4-53ae-4e45-832f-e5c29ca743b9","Type":"ContainerStarted","Data":"ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3"} Nov 04 00:57:47.303899 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:47.303760 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" event={"ID":"3603f9b4-53ae-4e45-832f-e5c29ca743b9","Type":"ContainerStarted","Data":"f947618ec4af545ba4640a197fa4a4ba8bcb571be9e5f7c09d91a26665a88d6e"} Nov 04 00:57:47.305385 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:47.305359 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" event={"ID":"e54204af-fcd5-442e-9009-f58ba803f3c4","Type":"ContainerStarted","Data":"077e4e228d7d48f11d7065b750b34ef53db74fdaf15323a0d6556ae35caaf76c"} Nov 04 00:57:47.305484 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:47.305405 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" event={"ID":"e54204af-fcd5-442e-9009-f58ba803f3c4","Type":"ContainerStarted","Data":"3679af22f4dd6a21469926a4b18d7c59851fca3d2edb01992667477f6f4c88d9"} Nov 04 00:57:47.306916 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:47.306893 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m84bl" event={"ID":"5b2c9231-5813-4143-93ca-2457278f37a2","Type":"ContainerStarted","Data":"4a111913a64983d56c0d747fdb74e1fe26dd0f7f17627a612b24ce7e5b6202e4"} Nov 04 00:57:47.372387 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:47.372359 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:57:48.312587 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:48.312539 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m84bl" event={"ID":"5b2c9231-5813-4143-93ca-2457278f37a2","Type":"ContainerStarted","Data":"7bf4ea67a7cdf929c5a744449ebdc703706daf715f4ade4cb75366d48e129eff"} Nov 04 00:57:48.313031 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:48.312611 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m84bl" event={"ID":"5b2c9231-5813-4143-93ca-2457278f37a2","Type":"ContainerStarted","Data":"61240da6ed681bb49cec84a148da109dd5731f95be58d330180c0037b1096d19"} Nov 04 00:57:48.313031 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:48.312565 5051 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:49.820523 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:49.820483 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:49.820925 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:49.820642 5051 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:49.824270 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:49.824251 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" Nov 04 00:57:49.856937 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:49.856907 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:49.857545 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:49.857524 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d024224-166d-4cb4-976f-75e0b47addae-service-ca-bundle\") pod \"router-default-669749fcf7-vqftg\" (UID: \"7d024224-166d-4cb4-976f-75e0b47addae\") " pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:49.939086 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:49.939049 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:49.957476 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:49.957443 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:49.959800 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:49.959771 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e7f2ccca-45df-41f8-bbf9-0b8aaa791160-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wtpbl\" (UID: \"e7f2ccca-45df-41f8-bbf9-0b8aaa791160\") " pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:50.228482 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:50.228387 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-669749fcf7-vqftg"] Nov 04 00:57:50.231110 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:50.231078 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d024224_166d_4cb4_976f_75e0b47addae.slice/crio-cf66192a5cb36be3f93efb576505809aa6b7c8ed7b78a1a1b278e0e7d21ea829 WatchSource:0}: Error finding container cf66192a5cb36be3f93efb576505809aa6b7c8ed7b78a1a1b278e0e7d21ea829: Status 404 returned error can't find the container with id cf66192a5cb36be3f93efb576505809aa6b7c8ed7b78a1a1b278e0e7d21ea829 Nov 04 00:57:50.255719 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:50.255696 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wtpbl" Nov 04 00:57:50.325977 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:50.325938 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-669749fcf7-vqftg" event={"ID":"7d024224-166d-4cb4-976f-75e0b47addae","Type":"ContainerStarted","Data":"ed854b131913d1174b092becbf74d58796cda75247fa7b7a4a8ba6c60bad238a"} Nov 04 00:57:50.326151 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:50.326004 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-669749fcf7-vqftg" event={"ID":"7d024224-166d-4cb4-976f-75e0b47addae","Type":"ContainerStarted","Data":"cf66192a5cb36be3f93efb576505809aa6b7c8ed7b78a1a1b278e0e7d21ea829"} Nov 04 00:57:50.398648 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:50.398599 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:57:50.531634 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:50.531601 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wtpbl"] Nov 04 00:57:50.534803 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:50.534767 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f2ccca_45df_41f8_bbf9_0b8aaa791160.slice/crio-bcdaa3a4815b35742c8b3dafedab6249497c23da41f1bad538c4b1c27f3d2dcf WatchSource:0}: Error finding container bcdaa3a4815b35742c8b3dafedab6249497c23da41f1bad538c4b1c27f3d2dcf: Status 404 returned error can't find the container with id bcdaa3a4815b35742c8b3dafedab6249497c23da41f1bad538c4b1c27f3d2dcf Nov 04 00:57:50.939641 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:50.939552 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:50.942283 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:50.942259 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:51.332249 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:51.332196 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtpbl" event={"ID":"e7f2ccca-45df-41f8-bbf9-0b8aaa791160","Type":"ContainerStarted","Data":"287c823347ce8813be62be24e8dc7c2837b0a639bacdf789659bc6bb6040fd25"} Nov 04 00:57:51.332408 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:51.332286 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtpbl" event={"ID":"e7f2ccca-45df-41f8-bbf9-0b8aaa791160","Type":"ContainerStarted","Data":"bcdaa3a4815b35742c8b3dafedab6249497c23da41f1bad538c4b1c27f3d2dcf"} Nov 04 00:57:52.010285 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:52.010256 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:57:52.010724 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:52.010461 5051 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:52.339401 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:52.339318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtpbl" event={"ID":"e7f2ccca-45df-41f8-bbf9-0b8aaa791160","Type":"ContainerStarted","Data":"255f7c38a65e4712bb230e9a978ef71a0b184dc8a09785e83b10371ff7467255"} Nov 04 00:57:52.339401 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:52.339388 5051 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:53.345951 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:53.345853 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wtpbl" event={"ID":"e7f2ccca-45df-41f8-bbf9-0b8aaa791160","Type":"ContainerStarted","Data":"76932fe566b637cc4fd62f0b1e4e5e8a004fe6676ee179c8c944ea11ac8d296e"} Nov 04 00:57:55.294248 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.294220 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mvtzk" Nov 04 00:57:55.369864 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.369836 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-czn8m-jxhgx"] Nov 04 00:57:55.374178 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.374154 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.386297 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.386271 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-czn8m-jxhgx"] Nov 04 00:57:55.430093 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.430059 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75c85b76dc-nfpkl"] Nov 04 00:57:55.431428 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.431399 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7"] Nov 04 00:57:55.437213 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.437189 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" Nov 04 00:57:55.449502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.448477 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jldt6\"" Nov 04 00:57:55.449502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.448572 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 04 00:57:55.449502 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.449475 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7"] Nov 04 00:57:55.506200 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.506112 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-trusted-ca-bundle\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.506375 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.506238 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn898\" (UniqueName: \"kubernetes.io/projected/552c2135-598a-4a6a-a71c-0d3fac0ecae4-kube-api-access-kn898\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.506375 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.506311 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-service-ca-bundle\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.506375 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.506336 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2da3265c-ae95-4220-a9d0-11ab44432db6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cpht7\" (UID: \"2da3265c-ae95-4220-a9d0-11ab44432db6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" Nov 04 00:57:55.506563 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.506397 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552c2135-598a-4a6a-a71c-0d3fac0ecae4-archives-path\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.511154 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.511109 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7wfdb"] Nov 04 00:57:55.514453 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.514433 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58c99c7dcd-fxv4f"] Nov 04 00:57:55.514570 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.514553 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:57:55.517543 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.517521 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b666b4847-lb44f"] Nov 04 00:57:55.517664 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.517554 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.520607 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.520588 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.524513 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.524492 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 04 00:57:55.524631 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.524545 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 04 00:57:55.534056 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.532943 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 04 00:57:55.534056 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.533106 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 04 00:57:55.534056 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.533233 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 04 00:57:55.534676 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.534650 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c99c7dcd-fxv4f"] Nov 04 00:57:55.536232 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.536213 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7wfdb"] Nov 04 00:57:55.538398 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.538376 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b666b4847-lb44f"] Nov 04 00:57:55.546642 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.546592 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-l7c6c\"" Nov 04 00:57:55.546737 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.546702 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 04 00:57:55.546775 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.546750 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 04 00:57:55.546835 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.546818 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q5sdf\"" Nov 04 00:57:55.607323 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607291 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-config\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.607513 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607343 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-serving-cert\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.607513 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607405 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112a671a-1e5f-4fe3-bc77-deb979fc4566-tmp\") pod \"downloads-6dd789c8b7-7wfdb\" (UID: \"112a671a-1e5f-4fe3-bc77-deb979fc4566\") " pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:57:55.607624 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607519 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-trusted-ca-bundle\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.607624 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607553 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn898\" (UniqueName: \"kubernetes.io/projected/552c2135-598a-4a6a-a71c-0d3fac0ecae4-kube-api-access-kn898\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.607624 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607579 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4145ecab-462f-4574-a346-61d7da286302-image-registry-private-configuration\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.607624 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607615 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4145ecab-462f-4574-a346-61d7da286302-trusted-ca\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.607823 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607764 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-service-ca-bundle\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.607823 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607801 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fm5k\" (UniqueName: \"kubernetes.io/projected/112a671a-1e5f-4fe3-bc77-deb979fc4566-kube-api-access-2fm5k\") pod \"downloads-6dd789c8b7-7wfdb\" (UID: \"112a671a-1e5f-4fe3-bc77-deb979fc4566\") " pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:57:55.608496 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608475 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-service-ca-bundle\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.608583 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.607837 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv5mq\" (UniqueName: \"kubernetes.io/projected/cb337f11-c1da-42c0-9b61-7d47d41113ed-kube-api-access-sv5mq\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.608667 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608610 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4145ecab-462f-4574-a346-61d7da286302-registry-certificates\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.608667 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608647 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2da3265c-ae95-4220-a9d0-11ab44432db6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cpht7\" (UID: \"2da3265c-ae95-4220-a9d0-11ab44432db6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" Nov 04 00:57:55.608766 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608675 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-oauth-config\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.608766 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608702 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-oauth-serving-cert\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.608852 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608778 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552c2135-598a-4a6a-a71c-0d3fac0ecae4-archives-path\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.608852 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608815 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-registry-tls\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.608933 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608856 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4145ecab-462f-4574-a346-61d7da286302-ca-trust-extracted\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.608933 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608852 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-trusted-ca-bundle\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.608933 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608872 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4145ecab-462f-4574-a346-61d7da286302-installation-pull-secrets\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.609037 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608947 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb337f11-c1da-42c0-9b61-7d47d41113ed-tmp\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.609037 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.608998 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-service-ca\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.609037 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.609021 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vplm2\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-kube-api-access-vplm2\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.609165 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.609053 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-bound-sa-token\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.609165 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.609060 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552c2135-598a-4a6a-a71c-0d3fac0ecae4-archives-path\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.611151 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.611112 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2da3265c-ae95-4220-a9d0-11ab44432db6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cpht7\" (UID: \"2da3265c-ae95-4220-a9d0-11ab44432db6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" Nov 04 00:57:55.618019 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.617994 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn898\" (UniqueName: \"kubernetes.io/projected/552c2135-598a-4a6a-a71c-0d3fac0ecae4-kube-api-access-kn898\") pod \"periodic-gathering-czn8m-jxhgx\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.688641 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.688608 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:57:55.710053 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710016 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-serving-cert\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.710053 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710052 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112a671a-1e5f-4fe3-bc77-deb979fc4566-tmp\") pod \"downloads-6dd789c8b7-7wfdb\" (UID: \"112a671a-1e5f-4fe3-bc77-deb979fc4566\") " pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:57:55.710254 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710076 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4145ecab-462f-4574-a346-61d7da286302-image-registry-private-configuration\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.710254 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710108 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4145ecab-462f-4574-a346-61d7da286302-trusted-ca\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.710254 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710144 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fm5k\" (UniqueName: \"kubernetes.io/projected/112a671a-1e5f-4fe3-bc77-deb979fc4566-kube-api-access-2fm5k\") pod \"downloads-6dd789c8b7-7wfdb\" (UID: \"112a671a-1e5f-4fe3-bc77-deb979fc4566\") " pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:57:55.710254 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710168 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv5mq\" (UniqueName: \"kubernetes.io/projected/cb337f11-c1da-42c0-9b61-7d47d41113ed-kube-api-access-sv5mq\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.710254 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710190 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4145ecab-462f-4574-a346-61d7da286302-registry-certificates\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.710254 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710219 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-oauth-config\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.710254 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710244 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-oauth-serving-cert\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.710625 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710291 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-registry-tls\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.710625 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710334 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4145ecab-462f-4574-a346-61d7da286302-ca-trust-extracted\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.710625 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710358 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4145ecab-462f-4574-a346-61d7da286302-installation-pull-secrets\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.710625 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710385 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb337f11-c1da-42c0-9b61-7d47d41113ed-tmp\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.710625 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710412 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-service-ca\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.710625 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710435 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vplm2\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-kube-api-access-vplm2\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.710625 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-bound-sa-token\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.710625 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112a671a-1e5f-4fe3-bc77-deb979fc4566-tmp\") pod \"downloads-6dd789c8b7-7wfdb\" (UID: \"112a671a-1e5f-4fe3-bc77-deb979fc4566\") " pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:57:55.711030 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.710720 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-config\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.711184 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.711157 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4145ecab-462f-4574-a346-61d7da286302-trusted-ca\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.711405 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.711380 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-oauth-serving-cert\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.711530 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.711495 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-config\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.712549 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.711767 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-service-ca\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.712549 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.711970 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb337f11-c1da-42c0-9b61-7d47d41113ed-tmp\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.712549 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.712179 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4145ecab-462f-4574-a346-61d7da286302-registry-certificates\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.712549 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.712350 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4145ecab-462f-4574-a346-61d7da286302-ca-trust-extracted\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.713401 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.713378 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4145ecab-462f-4574-a346-61d7da286302-image-registry-private-configuration\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.713799 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.713774 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-oauth-config\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.714258 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.714238 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-registry-tls\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.714670 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.714645 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4145ecab-462f-4574-a346-61d7da286302-installation-pull-secrets\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.715027 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.715011 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-serving-cert\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.723114 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.723089 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv5mq\" (UniqueName: \"kubernetes.io/projected/cb337f11-c1da-42c0-9b61-7d47d41113ed-kube-api-access-sv5mq\") pod \"console-58c99c7dcd-fxv4f\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.723483 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.723461 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-bound-sa-token\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.723584 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.723575 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fm5k\" (UniqueName: \"kubernetes.io/projected/112a671a-1e5f-4fe3-bc77-deb979fc4566-kube-api-access-2fm5k\") pod \"downloads-6dd789c8b7-7wfdb\" (UID: \"112a671a-1e5f-4fe3-bc77-deb979fc4566\") " pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:57:55.724372 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.724350 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vplm2\" (UniqueName: \"kubernetes.io/projected/4145ecab-462f-4574-a346-61d7da286302-kube-api-access-vplm2\") pod \"image-registry-b666b4847-lb44f\" (UID: \"4145ecab-462f-4574-a346-61d7da286302\") " pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.753791 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.753754 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" Nov 04 00:57:55.827414 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.827384 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:57:55.834110 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.834081 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:57:55.842619 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.842591 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:55.964807 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:55.964783 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-czn8m-jxhgx"] Nov 04 00:57:55.967310 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:55.967289 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552c2135_598a_4a6a_a71c_0d3fac0ecae4.slice/crio-522ac284fb5b92bc9da333359c8e175332c72e573abca4f8c6c10daf0f1901ef WatchSource:0}: Error finding container 522ac284fb5b92bc9da333359c8e175332c72e573abca4f8c6c10daf0f1901ef: Status 404 returned error can't find the container with id 522ac284fb5b92bc9da333359c8e175332c72e573abca4f8c6c10daf0f1901ef Nov 04 00:57:56.029309 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.029282 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7"] Nov 04 00:57:56.030824 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:56.030801 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da3265c_ae95_4220_a9d0_11ab44432db6.slice/crio-519885c9fbfa39c3e41a217a5b9175024b611f3e81b21d236fbad2368d95d56b WatchSource:0}: Error finding container 519885c9fbfa39c3e41a217a5b9175024b611f3e81b21d236fbad2368d95d56b: Status 404 returned error can't find the container with id 519885c9fbfa39c3e41a217a5b9175024b611f3e81b21d236fbad2368d95d56b Nov 04 00:57:56.109562 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.109522 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7wfdb"] Nov 04 00:57:56.111754 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:56.111720 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112a671a_1e5f_4fe3_bc77_deb979fc4566.slice/crio-f00f1d2a38d335f193e9716a125d35deac96b1b4d5847b06f5a87f9b5e635974 WatchSource:0}: Error finding container f00f1d2a38d335f193e9716a125d35deac96b1b4d5847b06f5a87f9b5e635974: Status 404 returned error can't find the container with id f00f1d2a38d335f193e9716a125d35deac96b1b4d5847b06f5a87f9b5e635974 Nov 04 00:57:56.122260 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.122233 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58c99c7dcd-fxv4f"] Nov 04 00:57:56.125602 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:56.125572 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb337f11_c1da_42c0_9b61_7d47d41113ed.slice/crio-a264997827dd87a7dad7b7ae2fd3d3a686513cf1c8739e782563e81fdd21dd8e WatchSource:0}: Error finding container a264997827dd87a7dad7b7ae2fd3d3a686513cf1c8739e782563e81fdd21dd8e: Status 404 returned error can't find the container with id a264997827dd87a7dad7b7ae2fd3d3a686513cf1c8739e782563e81fdd21dd8e Nov 04 00:57:56.163205 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.163175 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b666b4847-lb44f"] Nov 04 00:57:56.167107 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:56.167049 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4145ecab_462f_4574_a346_61d7da286302.slice/crio-5dcc493fa6db638105c6df3464b15e54f4dbc307bb530e57c991f5d9eaa95589 WatchSource:0}: Error finding container 5dcc493fa6db638105c6df3464b15e54f4dbc307bb530e57c991f5d9eaa95589: Status 404 returned error can't find the container with id 5dcc493fa6db638105c6df3464b15e54f4dbc307bb530e57c991f5d9eaa95589 Nov 04 00:57:56.360091 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.359994 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" event={"ID":"2da3265c-ae95-4220-a9d0-11ab44432db6","Type":"ContainerStarted","Data":"519885c9fbfa39c3e41a217a5b9175024b611f3e81b21d236fbad2368d95d56b"} Nov 04 00:57:56.361753 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.361726 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b666b4847-lb44f" event={"ID":"4145ecab-462f-4574-a346-61d7da286302","Type":"ContainerStarted","Data":"9cf8cf3310e32ea453f429b0431602118e653d7a9c353f2446afb885bd0a183d"} Nov 04 00:57:56.361885 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.361770 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b666b4847-lb44f" event={"ID":"4145ecab-462f-4574-a346-61d7da286302","Type":"ContainerStarted","Data":"5dcc493fa6db638105c6df3464b15e54f4dbc307bb530e57c991f5d9eaa95589"} Nov 04 00:57:56.361885 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.361820 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:57:56.363408 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.363382 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" event={"ID":"552c2135-598a-4a6a-a71c-0d3fac0ecae4","Type":"ContainerStarted","Data":"4d441ec047106ce5a4fc19d666c636966b0b209e69e24d4efbde9a668b6da886"} Nov 04 00:57:56.363513 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.363423 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" event={"ID":"552c2135-598a-4a6a-a71c-0d3fac0ecae4","Type":"ContainerStarted","Data":"522ac284fb5b92bc9da333359c8e175332c72e573abca4f8c6c10daf0f1901ef"} Nov 04 00:57:56.364601 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.364584 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c99c7dcd-fxv4f" event={"ID":"cb337f11-c1da-42c0-9b61-7d47d41113ed","Type":"ContainerStarted","Data":"a264997827dd87a7dad7b7ae2fd3d3a686513cf1c8739e782563e81fdd21dd8e"} Nov 04 00:57:56.365835 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.365816 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7wfdb" event={"ID":"112a671a-1e5f-4fe3-bc77-deb979fc4566","Type":"ContainerStarted","Data":"f00f1d2a38d335f193e9716a125d35deac96b1b4d5847b06f5a87f9b5e635974"} Nov 04 00:57:56.382183 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.382134 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b666b4847-lb44f" podStartSLOduration=1.382105766 podStartE2EDuration="1.382105766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:56.381414879 +0000 UTC m=+15.353163811" watchObservedRunningTime="2026-11-04 00:57:56.382105766 +0000 UTC m=+15.353854640" Nov 04 00:57:56.399242 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.399194 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" podStartSLOduration=1.3991794450000001 podStartE2EDuration="1.399179445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:56.398105243 +0000 UTC m=+15.369854125" watchObservedRunningTime="2026-11-04 00:57:56.399179445 +0000 UTC m=+15.370928315" Nov 04 00:57:56.818325 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.818247 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvtzk_b0807c2d-b756-42e8-86b3-97c30f23df72/dns/0.log" Nov 04 00:57:56.995149 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:56.995093 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvtzk_b0807c2d-b756-42e8-86b3-97c30f23df72/kube-rbac-proxy/0.log" Nov 04 00:57:57.395033 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:57.394924 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-shjsf_51ff9909-e56c-4d27-be31-574bf33a8f75/dns-node-resolver/0.log" Nov 04 00:57:57.597881 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:57.597825 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bfcf87bc8-zhslb_e54204af-fcd5-442e-9009-f58ba803f3c4/registry/0.log" Nov 04 00:57:57.794258 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:57.794216 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-75c85b76dc-nfpkl_3603f9b4-53ae-4e45-832f-e5c29ca743b9/registry/0.log" Nov 04 00:57:57.995180 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:57.995148 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b666b4847-lb44f_4145ecab-462f-4574-a346-61d7da286302/registry/0.log" Nov 04 00:57:58.019679 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.019648 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:58.019869 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.019802 5051 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 00:57:58.021640 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.021618 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-669749fcf7-vqftg" Nov 04 00:57:58.194284 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.194254 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcksb_5b7cc6df-efb4-4ed5-a299-99d3eafd8e56/node-ca/0.log" Nov 04 00:57:58.383382 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.383289 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" event={"ID":"2da3265c-ae95-4220-a9d0-11ab44432db6","Type":"ContainerStarted","Data":"f9b6b313c4bc1eb0fe87e28df4ead19987ad9da2a6c9a2a0582291d1ad81bb11"} Nov 04 00:57:58.383888 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.383845 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" Nov 04 00:57:58.389111 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.389087 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" Nov 04 00:57:58.400057 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.400006 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" podStartSLOduration=1.890264076 podStartE2EDuration="3.399993243s" totalImagesPullingTime="1.509729167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:57:58.398869647 +0000 UTC m=+17.370618518" watchObservedRunningTime="2026-11-04 00:57:58.399993243 +0000 UTC m=+17.371742114" Nov 04 00:57:58.594721 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.594690 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-669749fcf7-vqftg_7d024224-166d-4cb4-976f-75e0b47addae/router/0.log" Nov 04 00:57:58.663731 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.663656 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6578864c88-mqrw8"] Nov 04 00:57:58.667487 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.667466 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.675815 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.675709 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6578864c88-mqrw8"] Nov 04 00:57:58.678435 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.678411 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 04 00:57:58.744748 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.744709 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-service-ca\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.744928 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.744759 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-serving-cert\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.744928 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.744912 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flp79\" (UniqueName: \"kubernetes.io/projected/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-kube-api-access-flp79\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.745050 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.744962 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-tmp\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.745050 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.745037 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-oauth-serving-cert\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.745193 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.745107 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-oauth-config\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.745193 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.745160 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-trusted-ca-bundle\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.745297 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.745215 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-config\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.797050 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.796992 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vl6w6_8dbd7a56-ab21-413c-abac-36b1f79f452e/serve-healthcheck-canary/0.log" Nov 04 00:57:58.845822 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.845787 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-config\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.845994 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.845928 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-service-ca\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.845994 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.845960 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-serving-cert\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.846114 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.846018 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flp79\" (UniqueName: \"kubernetes.io/projected/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-kube-api-access-flp79\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.846114 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.846047 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-tmp\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.846114 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.846074 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-oauth-serving-cert\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.846114 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.846111 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-oauth-config\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.846351 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.846158 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-trusted-ca-bundle\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.846660 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.846641 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-config\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.846978 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.846957 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-trusted-ca-bundle\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.847080 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.847035 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-tmp\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.847495 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.847472 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-oauth-serving-cert\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.847604 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.847576 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-service-ca\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.855151 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.850049 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-serving-cert\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.855151 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.850228 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-oauth-config\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.855349 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.855249 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flp79\" (UniqueName: \"kubernetes.io/projected/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-kube-api-access-flp79\") pod \"console-6578864c88-mqrw8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:58.982417 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:58.982362 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:57:59.269904 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.269832 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6578864c88-mqrw8"] Nov 04 00:57:59.273357 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:57:59.273285 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8dbde1_4579_4f9a_9e64_8fa5ad980ec8.slice/crio-490af450554fdf778fa2bc003715cfa279900bc38ac61a881a9afdadd4edbafd WatchSource:0}: Error finding container 490af450554fdf778fa2bc003715cfa279900bc38ac61a881a9afdadd4edbafd: Status 404 returned error can't find the container with id 490af450554fdf778fa2bc003715cfa279900bc38ac61a881a9afdadd4edbafd Nov 04 00:57:59.393629 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.393587 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6578864c88-mqrw8" event={"ID":"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8","Type":"ContainerStarted","Data":"490af450554fdf778fa2bc003715cfa279900bc38ac61a881a9afdadd4edbafd"} Nov 04 00:57:59.417846 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.417811 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-q5894"] Nov 04 00:57:59.426812 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.426784 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.431226 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.429348 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-q5894"] Nov 04 00:57:59.433043 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.432331 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 04 00:57:59.433043 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.432806 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 04 00:57:59.433043 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.432925 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 00:57:59.433043 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.433015 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-89fbq\"" Nov 04 00:57:59.551892 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.551809 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526a942d-b2a6-42f6-a48f-761bdb2f73fd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.551892 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.551878 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggs22\" (UniqueName: \"kubernetes.io/projected/526a942d-b2a6-42f6-a48f-761bdb2f73fd-kube-api-access-ggs22\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.552099 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.551916 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/526a942d-b2a6-42f6-a48f-761bdb2f73fd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.552099 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.551946 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/526a942d-b2a6-42f6-a48f-761bdb2f73fd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.653244 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.653215 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526a942d-b2a6-42f6-a48f-761bdb2f73fd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.653379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.653273 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggs22\" (UniqueName: \"kubernetes.io/projected/526a942d-b2a6-42f6-a48f-761bdb2f73fd-kube-api-access-ggs22\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.653379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.653311 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/526a942d-b2a6-42f6-a48f-761bdb2f73fd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.653379 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.653345 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/526a942d-b2a6-42f6-a48f-761bdb2f73fd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.654149 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.654073 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/526a942d-b2a6-42f6-a48f-761bdb2f73fd-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.656042 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.656016 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/526a942d-b2a6-42f6-a48f-761bdb2f73fd-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.656223 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.656202 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/526a942d-b2a6-42f6-a48f-761bdb2f73fd-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.661693 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.661669 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggs22\" (UniqueName: \"kubernetes.io/projected/526a942d-b2a6-42f6-a48f-761bdb2f73fd-kube-api-access-ggs22\") pod \"prometheus-operator-9cd89b74c-q5894\" (UID: \"526a942d-b2a6-42f6-a48f-761bdb2f73fd\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:57:59.749016 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:57:59.748980 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" Nov 04 00:58:00.040642 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:00.040622 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-q5894"] Nov 04 00:58:00.043383 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:58:00.043345 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526a942d_b2a6_42f6_a48f_761bdb2f73fd.slice/crio-6e56400ba0dae2882a2c12b4f6e97d0a20b6c70a36d7231d6134df7eab280d01 WatchSource:0}: Error finding container 6e56400ba0dae2882a2c12b4f6e97d0a20b6c70a36d7231d6134df7eab280d01: Status 404 returned error can't find the container with id 6e56400ba0dae2882a2c12b4f6e97d0a20b6c70a36d7231d6134df7eab280d01 Nov 04 00:58:00.402446 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:00.402405 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c99c7dcd-fxv4f" event={"ID":"cb337f11-c1da-42c0-9b61-7d47d41113ed","Type":"ContainerStarted","Data":"aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e"} Nov 04 00:58:00.404147 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:00.404106 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6578864c88-mqrw8" event={"ID":"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8","Type":"ContainerStarted","Data":"4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e"} Nov 04 00:58:00.405450 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:00.405383 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" event={"ID":"526a942d-b2a6-42f6-a48f-761bdb2f73fd","Type":"ContainerStarted","Data":"6e56400ba0dae2882a2c12b4f6e97d0a20b6c70a36d7231d6134df7eab280d01"} Nov 04 00:58:00.423397 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:00.423303 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58c99c7dcd-fxv4f" podStartSLOduration=1.934315975 podStartE2EDuration="5.423285157s" totalImagesPullingTime="3.488969182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:00.422467847 +0000 UTC m=+19.394216731" watchObservedRunningTime="2026-11-04 00:58:00.423285157 +0000 UTC m=+19.395034033" Nov 04 00:58:02.417186 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:02.417145 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" event={"ID":"526a942d-b2a6-42f6-a48f-761bdb2f73fd","Type":"ContainerStarted","Data":"4ad855e84bae7f8c3e8e5f9d1e185b6cf98f750714156e63bd07995152c68228"} Nov 04 00:58:02.417680 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:02.417247 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" event={"ID":"526a942d-b2a6-42f6-a48f-761bdb2f73fd","Type":"ContainerStarted","Data":"0ca9ebbe34c823b546c277b0dc30884f0d777ea572b3fdab5c96b4c9010d15f0"} Nov 04 00:58:02.437646 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:02.437595 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-q5894" podStartSLOduration=1.809998875 podStartE2EDuration="3.437581113s" totalImagesPullingTime="1.627582238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:02.436430151 +0000 UTC m=+21.408179022" watchObservedRunningTime="2026-11-04 00:58:02.437581113 +0000 UTC m=+21.409329979" Nov 04 00:58:02.438441 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:02.438400 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6578864c88-mqrw8" podStartSLOduration=3.9837400560000003 podStartE2EDuration="4.438385809s" totalImagesPullingTime="454.645753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:00.44726643 +0000 UTC m=+19.419015301" watchObservedRunningTime="2026-11-04 00:58:02.438385809 +0000 UTC m=+21.410134681" Nov 04 00:58:04.107860 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:58:04.107825 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 00:58:04.989663 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:04.989619 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p52db"] Nov 04 00:58:05.024107 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.023306 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.028710 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.028530 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 00:58:05.028710 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.028576 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 00:58:05.028710 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.028629 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 00:58:05.028710 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.028677 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5llll\"" Nov 04 00:58:05.117310 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117270 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4850566-c180-40ed-b33a-45648462327d-metrics-client-ca\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.117922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117318 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c4850566-c180-40ed-b33a-45648462327d-node-exporter-accelerators-collector-config\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.117922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117439 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qph2j\" (UniqueName: \"kubernetes.io/projected/c4850566-c180-40ed-b33a-45648462327d-kube-api-access-qph2j\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.117922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117507 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c4850566-c180-40ed-b33a-45648462327d-node-exporter-textfile\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.117922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117547 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c4850566-c180-40ed-b33a-45648462327d-node-exporter-tls\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.117922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117649 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-sys\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.117922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117683 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-root\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.117922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117771 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4850566-c180-40ed-b33a-45648462327d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.117922 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.117812 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-node-exporter-wtmp\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.218554 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218522 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-node-exporter-wtmp\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.218742 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218568 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4850566-c180-40ed-b33a-45648462327d-metrics-client-ca\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.218742 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218594 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c4850566-c180-40ed-b33a-45648462327d-node-exporter-accelerators-collector-config\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.218742 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218625 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qph2j\" (UniqueName: \"kubernetes.io/projected/c4850566-c180-40ed-b33a-45648462327d-kube-api-access-qph2j\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.218742 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218658 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c4850566-c180-40ed-b33a-45648462327d-node-exporter-textfile\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.218742 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218692 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c4850566-c180-40ed-b33a-45648462327d-node-exporter-tls\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.218742 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218714 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-node-exporter-wtmp\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.219108 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218765 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-sys\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.219108 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218793 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-root\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.219108 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.218834 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4850566-c180-40ed-b33a-45648462327d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.219287 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.219144 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-sys\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.219287 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.219169 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c4850566-c180-40ed-b33a-45648462327d-node-exporter-textfile\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.219287 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.219211 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c4850566-c180-40ed-b33a-45648462327d-root\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.219287 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.219280 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4850566-c180-40ed-b33a-45648462327d-metrics-client-ca\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.219473 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.219362 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c4850566-c180-40ed-b33a-45648462327d-node-exporter-accelerators-collector-config\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.222316 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.222276 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c4850566-c180-40ed-b33a-45648462327d-node-exporter-tls\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.222500 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.222479 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4850566-c180-40ed-b33a-45648462327d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.227988 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.227965 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qph2j\" (UniqueName: \"kubernetes.io/projected/c4850566-c180-40ed-b33a-45648462327d-kube-api-access-qph2j\") pod \"node-exporter-p52db\" (UID: \"c4850566-c180-40ed-b33a-45648462327d\") " pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.382248 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.382152 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p52db" Nov 04 00:58:05.426484 ip-10-0-137-48 kubenswrapper[5051]: W1104 00:58:05.426449 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4850566_c180_40ed_b33a_45648462327d.slice/crio-d2d4abeeb195822b445c551aff6ba6fb8fa1caacf14e4de48628850db05d383a WatchSource:0}: Error finding container d2d4abeeb195822b445c551aff6ba6fb8fa1caacf14e4de48628850db05d383a: Status 404 returned error can't find the container with id d2d4abeeb195822b445c551aff6ba6fb8fa1caacf14e4de48628850db05d383a Nov 04 00:58:05.435345 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.435311 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p52db" event={"ID":"c4850566-c180-40ed-b33a-45648462327d","Type":"ContainerStarted","Data":"d2d4abeeb195822b445c551aff6ba6fb8fa1caacf14e4de48628850db05d383a"} Nov 04 00:58:05.437536 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.437516 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:58:05.673776 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.673704 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w6tt8" Nov 04 00:58:05.834707 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.834672 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:58:05.834861 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.834737 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:58:05.840549 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:05.840522 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:58:06.445180 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:06.445147 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:58:08.452423 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:08.452387 5051 generic.go:356] "Generic (PLEG): container finished" podID="c4850566-c180-40ed-b33a-45648462327d" containerID="eca9d9eddd7b47524f289ed2a0258078f165dc1591e8fd3f2a09ff6a59e43365" exitCode=0 Nov 04 00:58:08.452872 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:08.452493 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p52db" event={"ID":"c4850566-c180-40ed-b33a-45648462327d","Type":"ContainerDied","Data":"eca9d9eddd7b47524f289ed2a0258078f165dc1591e8fd3f2a09ff6a59e43365"} Nov 04 00:58:08.983426 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:08.983387 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:58:08.983603 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:08.983471 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:58:08.989522 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:08.989492 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:58:09.322868 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:09.322793 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:58:09.460461 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:09.460424 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p52db" event={"ID":"c4850566-c180-40ed-b33a-45648462327d","Type":"ContainerStarted","Data":"4e38588d7321ea17525ea8beca90651da390794809dd348f78a0bf5cd7e586c6"} Nov 04 00:58:09.460956 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:09.460483 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p52db" event={"ID":"c4850566-c180-40ed-b33a-45648462327d","Type":"ContainerStarted","Data":"d0ccd22a0b205059fecc47bc5f8df3d75c597b3ea0ddb322217ddd7869d5f22d"} Nov 04 00:58:09.465219 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:09.465193 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:58:09.487566 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:09.487501 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p52db" podStartSLOduration=3.105017294 podStartE2EDuration="5.487482549s" totalImagesPullingTime="2.282287426s" totalInitContainerRuntime="100.177829ms" isStatefulPod=true podCreationTimestamp="2026-11-04 00:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:09.486253457 +0000 UTC m=+28.458002327" watchObservedRunningTime="2026-11-04 00:58:09.487482549 +0000 UTC m=+28.459231424" Nov 04 00:58:09.583766 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:09.583693 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c99c7dcd-fxv4f"] Nov 04 00:58:10.091821 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:10.091781 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" Nov 04 00:58:14.488723 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:14.488647 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7wfdb" event={"ID":"112a671a-1e5f-4fe3-bc77-deb979fc4566","Type":"ContainerStarted","Data":"758d204c3aac53b99f85cca5db4b7ac8ee7c124abf92cd5bbb5b8af11f80bcf8"} Nov 04 00:58:14.489307 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:14.489281 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:58:14.503199 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:14.503151 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7wfdb" Nov 04 00:58:14.535898 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:14.535834 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-7wfdb" podStartSLOduration=1.408890847 podStartE2EDuration="19.535815161s" totalImagesPullingTime="18.126924314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 00:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 00:58:14.53255901 +0000 UTC m=+33.504307894" watchObservedRunningTime="2026-11-04 00:58:14.535815161 +0000 UTC m=+33.507564035" Nov 04 00:58:15.522347 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:15.522316 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zh5bh" Nov 04 00:58:17.381771 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:17.381744 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b666b4847-lb44f" Nov 04 00:58:17.435860 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:17.435810 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bfcf87bc8-zhslb"] Nov 04 00:58:20.454908 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.454840 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" podUID="3603f9b4-53ae-4e45-832f-e5c29ca743b9" containerName="registry" containerID="cri-o://ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3" gracePeriod=30 Nov 04 00:58:20.732454 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.732431 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:58:20.791252 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791219 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token\") pod \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " Nov 04 00:58:20.791489 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791270 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates\") pod \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " Nov 04 00:58:20.791489 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791305 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration\") pod \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " Nov 04 00:58:20.791489 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791357 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") pod \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " Nov 04 00:58:20.791489 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791400 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted\") pod \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " Nov 04 00:58:20.791489 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791437 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca\") pod \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " Nov 04 00:58:20.791489 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791467 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets\") pod \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " Nov 04 00:58:20.792202 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791593 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462\") pod \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\" (UID: \"3603f9b4-53ae-4e45-832f-e5c29ca743b9\") " Nov 04 00:58:20.792202 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.791863 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates" pod "3603f9b4-53ae-4e45-832f-e5c29ca743b9" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:20.792460 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.792399 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca" pod "3603f9b4-53ae-4e45-832f-e5c29ca743b9" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:20.795514 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.795482 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462" pod "3603f9b4-53ae-4e45-832f-e5c29ca743b9" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9"). InnerVolumeSpecName "kube-api-access-dc462". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:20.798615 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.798586 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls" pod "3603f9b4-53ae-4e45-832f-e5c29ca743b9" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:20.800565 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.800524 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration" pod "3603f9b4-53ae-4e45-832f-e5c29ca743b9" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:20.800652 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.800561 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token" pod "3603f9b4-53ae-4e45-832f-e5c29ca743b9" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:20.800714 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.800651 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets" pod "3603f9b4-53ae-4e45-832f-e5c29ca743b9" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:20.803986 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.803952 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted" pod "3603f9b4-53ae-4e45-832f-e5c29ca743b9" (UID: "3603f9b4-53ae-4e45-832f-e5c29ca743b9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:20.892786 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.892753 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc462\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-kube-api-access-dc462\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:20.892909 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.892791 5051 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-bound-sa-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:20.892909 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.892806 5051 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-certificates\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:20.892909 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.892817 5051 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-image-registry-private-configuration\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:20.892909 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.892826 5051 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3603f9b4-53ae-4e45-832f-e5c29ca743b9-registry-tls\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:20.892909 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.892835 5051 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3603f9b4-53ae-4e45-832f-e5c29ca743b9-ca-trust-extracted\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:20.892909 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.892847 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3603f9b4-53ae-4e45-832f-e5c29ca743b9-trusted-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:20.892909 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:20.892860 5051 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3603f9b4-53ae-4e45-832f-e5c29ca743b9-installation-pull-secrets\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:21.256043 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.256012 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6578864c88-mqrw8"] Nov 04 00:58:21.544806 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.544767 5051 generic.go:356] "Generic (PLEG): container finished" podID="3603f9b4-53ae-4e45-832f-e5c29ca743b9" containerID="ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3" exitCode=0 Nov 04 00:58:21.545385 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.544811 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" Nov 04 00:58:21.545385 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.544834 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" event={"ID":"3603f9b4-53ae-4e45-832f-e5c29ca743b9","Type":"ContainerDied","Data":"ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3"} Nov 04 00:58:21.545385 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.544881 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75c85b76dc-nfpkl" event={"ID":"3603f9b4-53ae-4e45-832f-e5c29ca743b9","Type":"ContainerDied","Data":"f947618ec4af545ba4640a197fa4a4ba8bcb571be9e5f7c09d91a26665a88d6e"} Nov 04 00:58:21.545385 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.544908 5051 scope.go:122] "RemoveContainer" containerID="ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3" Nov 04 00:58:21.555557 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.555537 5051 scope.go:122] "RemoveContainer" containerID="ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3" Nov 04 00:58:21.555870 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:58:21.555839 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3\": container with ID starting with ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3 not found: ID does not exist" containerID="ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3" Nov 04 00:58:21.555973 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.555883 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3"} err="failed to get container status \"ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3\": rpc error: code = NotFound desc = could not find container \"ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3\": container with ID starting with ae5ad3418aeb09680ec501cfcf4a320dfe9cb32a81ec69ce8093f79f51e7f4e3 not found: ID does not exist" Nov 04 00:58:21.563863 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.563841 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75c85b76dc-nfpkl"] Nov 04 00:58:21.567549 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:21.567528 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75c85b76dc-nfpkl"] Nov 04 00:58:23.185581 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:23.185545 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3603f9b4-53ae-4e45-832f-e5c29ca743b9" path="/var/lib/kubelet/pods/3603f9b4-53ae-4e45-832f-e5c29ca743b9/volumes" Nov 04 00:58:28.851331 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:28.851298 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-tw9hp-q2fgs"] Nov 04 00:58:28.851878 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:28.851511 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" podUID="c9027aeb-9f01-4ef8-b936-15c9382b2af8" containerName="insights-gathering" containerID="cri-o://2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab" gracePeriod=30 Nov 04 00:58:29.198070 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.198043 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:58:29.262203 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.262172 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path\") pod \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " Nov 04 00:58:29.262339 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.262233 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle\") pod \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " Nov 04 00:58:29.262339 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.262305 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle\") pod \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " Nov 04 00:58:29.262419 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.262344 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8\" (UniqueName: \"kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8\") pod \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\" (UID: \"c9027aeb-9f01-4ef8-b936-15c9382b2af8\") " Nov 04 00:58:29.262560 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.262505 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle" pod "c9027aeb-9f01-4ef8-b936-15c9382b2af8" (UID: "c9027aeb-9f01-4ef8-b936-15c9382b2af8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:29.262636 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.262577 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path" pod "c9027aeb-9f01-4ef8-b936-15c9382b2af8" (UID: "c9027aeb-9f01-4ef8-b936-15c9382b2af8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:29.262636 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.262613 5051 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-service-ca-bundle\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:29.262737 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.262650 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle" pod "c9027aeb-9f01-4ef8-b936-15c9382b2af8" (UID: "c9027aeb-9f01-4ef8-b936-15c9382b2af8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:29.265154 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.265113 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8" pod "c9027aeb-9f01-4ef8-b936-15c9382b2af8" (UID: "c9027aeb-9f01-4ef8-b936-15c9382b2af8"). InnerVolumeSpecName "kube-api-access-mnff8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:29.363902 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.363874 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnff8\" (UniqueName: \"kubernetes.io/projected/c9027aeb-9f01-4ef8-b936-15c9382b2af8-kube-api-access-mnff8\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:29.363902 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.363899 5051 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9027aeb-9f01-4ef8-b936-15c9382b2af8-archives-path\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:29.364052 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.363909 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9027aeb-9f01-4ef8-b936-15c9382b2af8-trusted-ca-bundle\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:29.583177 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.582857 5051 generic.go:356] "Generic (PLEG): container finished" podID="c9027aeb-9f01-4ef8-b936-15c9382b2af8" containerID="2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab" exitCode=2 Nov 04 00:58:29.583177 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.582925 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" event={"ID":"c9027aeb-9f01-4ef8-b936-15c9382b2af8","Type":"ContainerDied","Data":"2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab"} Nov 04 00:58:29.583177 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.582971 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" event={"ID":"c9027aeb-9f01-4ef8-b936-15c9382b2af8","Type":"ContainerDied","Data":"0a940fb409c7170d592362e00ec0c2beadf3a50c2102d26c10bd1c05dd6dad67"} Nov 04 00:58:29.583177 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.582996 5051 scope.go:122] "RemoveContainer" containerID="2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab" Nov 04 00:58:29.583713 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.583150 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tw9hp-q2fgs" Nov 04 00:58:29.594610 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.594589 5051 scope.go:122] "RemoveContainer" containerID="2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab" Nov 04 00:58:29.594896 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:58:29.594874 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab\": container with ID starting with 2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab not found: ID does not exist" containerID="2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab" Nov 04 00:58:29.594966 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.594908 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab"} err="failed to get container status \"2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab\": rpc error: code = NotFound desc = could not find container \"2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab\": container with ID starting with 2cac1b929ab6b062893b3fddafb251e6055c5d69e9dbf48634df8b3b785c8aab not found: ID does not exist" Nov 04 00:58:29.614161 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.614136 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-tw9hp-q2fgs"] Nov 04 00:58:29.616059 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:29.616034 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-tw9hp-q2fgs"] Nov 04 00:58:31.185652 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:31.185618 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9027aeb-9f01-4ef8-b936-15c9382b2af8" path="/var/lib/kubelet/pods/c9027aeb-9f01-4ef8-b936-15c9382b2af8/volumes" Nov 04 00:58:34.104581 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:58:34.104540 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 00:58:34.612183 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:34.612099 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58c99c7dcd-fxv4f" podUID="cb337f11-c1da-42c0-9b61-7d47d41113ed" containerName="console" containerID="cri-o://aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e" gracePeriod=15 Nov 04 00:58:34.908334 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:34.908309 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:58:35.014685 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.014647 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-serving-cert\") pod \"cb337f11-c1da-42c0-9b61-7d47d41113ed\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " Nov 04 00:58:35.014877 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.014694 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-oauth-serving-cert\") pod \"cb337f11-c1da-42c0-9b61-7d47d41113ed\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " Nov 04 00:58:35.014877 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.014741 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb337f11-c1da-42c0-9b61-7d47d41113ed-kube-api-access-sv5mq\" (UniqueName: \"kubernetes.io/projected/cb337f11-c1da-42c0-9b61-7d47d41113ed-kube-api-access-sv5mq\") pod \"cb337f11-c1da-42c0-9b61-7d47d41113ed\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " Nov 04 00:58:35.014877 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.014767 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-oauth-config\") pod \"cb337f11-c1da-42c0-9b61-7d47d41113ed\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " Nov 04 00:58:35.014877 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.014848 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-config\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-config\") pod \"cb337f11-c1da-42c0-9b61-7d47d41113ed\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " Nov 04 00:58:35.015062 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.014903 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb337f11-c1da-42c0-9b61-7d47d41113ed-tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb337f11-c1da-42c0-9b61-7d47d41113ed-tmp\") pod \"cb337f11-c1da-42c0-9b61-7d47d41113ed\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " Nov 04 00:58:35.015062 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.014930 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-service-ca\") pod \"cb337f11-c1da-42c0-9b61-7d47d41113ed\" (UID: \"cb337f11-c1da-42c0-9b61-7d47d41113ed\") " Nov 04 00:58:35.015299 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.015260 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb337f11-c1da-42c0-9b61-7d47d41113ed-tmp" pod "cb337f11-c1da-42c0-9b61-7d47d41113ed" (UID: "cb337f11-c1da-42c0-9b61-7d47d41113ed"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:35.015299 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.015287 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-oauth-serving-cert" pod "cb337f11-c1da-42c0-9b61-7d47d41113ed" (UID: "cb337f11-c1da-42c0-9b61-7d47d41113ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:35.015456 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.015404 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-config" pod "cb337f11-c1da-42c0-9b61-7d47d41113ed" (UID: "cb337f11-c1da-42c0-9b61-7d47d41113ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:35.015517 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.015489 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-service-ca" pod "cb337f11-c1da-42c0-9b61-7d47d41113ed" (UID: "cb337f11-c1da-42c0-9b61-7d47d41113ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:35.017041 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.017017 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-serving-cert" pod "cb337f11-c1da-42c0-9b61-7d47d41113ed" (UID: "cb337f11-c1da-42c0-9b61-7d47d41113ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:35.017161 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.017054 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb337f11-c1da-42c0-9b61-7d47d41113ed-kube-api-access-sv5mq" pod "cb337f11-c1da-42c0-9b61-7d47d41113ed" (UID: "cb337f11-c1da-42c0-9b61-7d47d41113ed"). InnerVolumeSpecName "kube-api-access-sv5mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:35.017161 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.017104 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-oauth-config" pod "cb337f11-c1da-42c0-9b61-7d47d41113ed" (UID: "cb337f11-c1da-42c0-9b61-7d47d41113ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:35.116393 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.116354 5051 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-config\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:35.116393 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.116386 5051 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb337f11-c1da-42c0-9b61-7d47d41113ed-tmp\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:35.116393 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.116396 5051 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:35.116869 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.116407 5051 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-serving-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:35.116869 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.116417 5051 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cb337f11-c1da-42c0-9b61-7d47d41113ed-oauth-serving-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:35.116869 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.116425 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv5mq\" (UniqueName: \"kubernetes.io/projected/cb337f11-c1da-42c0-9b61-7d47d41113ed-kube-api-access-sv5mq\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:35.116869 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.116434 5051 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cb337f11-c1da-42c0-9b61-7d47d41113ed-console-oauth-config\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:35.610582 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.610547 5051 generic.go:356] "Generic (PLEG): container finished" podID="cb337f11-c1da-42c0-9b61-7d47d41113ed" containerID="aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e" exitCode=0 Nov 04 00:58:35.610832 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.610597 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58c99c7dcd-fxv4f" Nov 04 00:58:35.610832 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.610632 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c99c7dcd-fxv4f" event={"ID":"cb337f11-c1da-42c0-9b61-7d47d41113ed","Type":"ContainerDied","Data":"aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e"} Nov 04 00:58:35.610832 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.610686 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58c99c7dcd-fxv4f" event={"ID":"cb337f11-c1da-42c0-9b61-7d47d41113ed","Type":"ContainerDied","Data":"a264997827dd87a7dad7b7ae2fd3d3a686513cf1c8739e782563e81fdd21dd8e"} Nov 04 00:58:35.610832 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.610711 5051 scope.go:122] "RemoveContainer" containerID="aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e" Nov 04 00:58:35.619420 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.619233 5051 scope.go:122] "RemoveContainer" containerID="aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e" Nov 04 00:58:35.619536 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:58:35.619517 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e\": container with ID starting with aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e not found: ID does not exist" containerID="aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e" Nov 04 00:58:35.619587 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.619544 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e"} err="failed to get container status \"aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e\": rpc error: code = NotFound desc = could not find container \"aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e\": container with ID starting with aa81c833de087df5d41d6bfbd8e6eb4e5eda1d7a5bc5d9e7422e8c6696e3306e not found: ID does not exist" Nov 04 00:58:35.626518 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.626495 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58c99c7dcd-fxv4f"] Nov 04 00:58:35.628175 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:35.628154 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58c99c7dcd-fxv4f"] Nov 04 00:58:37.189332 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:37.189298 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb337f11-c1da-42c0-9b61-7d47d41113ed" path="/var/lib/kubelet/pods/cb337f11-c1da-42c0-9b61-7d47d41113ed/volumes" Nov 04 00:58:42.481157 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.481084 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" podUID="e54204af-fcd5-442e-9009-f58ba803f3c4" containerName="registry" containerID="cri-o://077e4e228d7d48f11d7065b750b34ef53db74fdaf15323a0d6556ae35caaf76c" gracePeriod=30 Nov 04 00:58:42.642422 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.642390 5051 generic.go:356] "Generic (PLEG): container finished" podID="e54204af-fcd5-442e-9009-f58ba803f3c4" containerID="077e4e228d7d48f11d7065b750b34ef53db74fdaf15323a0d6556ae35caaf76c" exitCode=0 Nov 04 00:58:42.642546 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.642459 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" event={"ID":"e54204af-fcd5-442e-9009-f58ba803f3c4","Type":"ContainerDied","Data":"077e4e228d7d48f11d7065b750b34ef53db74fdaf15323a0d6556ae35caaf76c"} Nov 04 00:58:42.767481 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.767452 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:58:42.882064 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882034 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") pod \"e54204af-fcd5-442e-9009-f58ba803f3c4\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " Nov 04 00:58:42.882227 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882070 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted\") pod \"e54204af-fcd5-442e-9009-f58ba803f3c4\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " Nov 04 00:58:42.882227 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882095 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets\") pod \"e54204af-fcd5-442e-9009-f58ba803f3c4\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " Nov 04 00:58:42.882227 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882114 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz\") pod \"e54204af-fcd5-442e-9009-f58ba803f3c4\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " Nov 04 00:58:42.882227 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882179 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration\") pod \"e54204af-fcd5-442e-9009-f58ba803f3c4\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " Nov 04 00:58:42.882227 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882220 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates\") pod \"e54204af-fcd5-442e-9009-f58ba803f3c4\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " Nov 04 00:58:42.882539 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882270 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token\") pod \"e54204af-fcd5-442e-9009-f58ba803f3c4\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " Nov 04 00:58:42.882539 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882310 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca\") pod \"e54204af-fcd5-442e-9009-f58ba803f3c4\" (UID: \"e54204af-fcd5-442e-9009-f58ba803f3c4\") " Nov 04 00:58:42.882897 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.882863 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca" pod "e54204af-fcd5-442e-9009-f58ba803f3c4" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:42.884971 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.884902 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz" pod "e54204af-fcd5-442e-9009-f58ba803f3c4" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4"). InnerVolumeSpecName "kube-api-access-h7xnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:42.885262 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.885229 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls" pod "e54204af-fcd5-442e-9009-f58ba803f3c4" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:42.885405 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.885374 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets" pod "e54204af-fcd5-442e-9009-f58ba803f3c4" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:42.885536 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.885473 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration" pod "e54204af-fcd5-442e-9009-f58ba803f3c4" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:42.885607 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.885546 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token" pod "e54204af-fcd5-442e-9009-f58ba803f3c4" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:42.886386 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.886361 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates" pod "e54204af-fcd5-442e-9009-f58ba803f3c4" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:42.891461 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.891440 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted" pod "e54204af-fcd5-442e-9009-f58ba803f3c4" (UID: "e54204af-fcd5-442e-9009-f58ba803f3c4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:42.983723 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.983665 5051 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-bound-sa-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:42.983723 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.983688 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-trusted-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:42.983723 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.983698 5051 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-tls\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:42.983723 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.983708 5051 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e54204af-fcd5-442e-9009-f58ba803f3c4-ca-trust-extracted\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:42.983723 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.983717 5051 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-installation-pull-secrets\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:42.983723 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.983726 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7xnz\" (UniqueName: \"kubernetes.io/projected/e54204af-fcd5-442e-9009-f58ba803f3c4-kube-api-access-h7xnz\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:42.983989 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.983735 5051 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e54204af-fcd5-442e-9009-f58ba803f3c4-image-registry-private-configuration\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:42.983989 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:42.983744 5051 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e54204af-fcd5-442e-9009-f58ba803f3c4-registry-certificates\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:43.647616 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:43.647587 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" Nov 04 00:58:43.648021 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:43.647589 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bfcf87bc8-zhslb" event={"ID":"e54204af-fcd5-442e-9009-f58ba803f3c4","Type":"ContainerDied","Data":"3679af22f4dd6a21469926a4b18d7c59851fca3d2edb01992667477f6f4c88d9"} Nov 04 00:58:43.648021 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:43.647719 5051 scope.go:122] "RemoveContainer" containerID="077e4e228d7d48f11d7065b750b34ef53db74fdaf15323a0d6556ae35caaf76c" Nov 04 00:58:43.664712 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:43.664692 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bfcf87bc8-zhslb"] Nov 04 00:58:43.670987 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:43.670964 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bfcf87bc8-zhslb"] Nov 04 00:58:45.186674 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:45.186640 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54204af-fcd5-442e-9009-f58ba803f3c4" path="/var/lib/kubelet/pods/e54204af-fcd5-442e-9009-f58ba803f3c4/volumes" Nov 04 00:58:46.280892 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.280835 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6578864c88-mqrw8" podUID="1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" containerName="console" containerID="cri-o://4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e" gracePeriod=15 Nov 04 00:58:46.581459 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.581435 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:58:46.661376 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.661343 5051 generic.go:356] "Generic (PLEG): container finished" podID="1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" containerID="4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e" exitCode=0 Nov 04 00:58:46.661529 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.661379 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6578864c88-mqrw8" event={"ID":"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8","Type":"ContainerDied","Data":"4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e"} Nov 04 00:58:46.661529 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.661386 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6578864c88-mqrw8" Nov 04 00:58:46.661529 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.661414 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6578864c88-mqrw8" event={"ID":"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8","Type":"ContainerDied","Data":"490af450554fdf778fa2bc003715cfa279900bc38ac61a881a9afdadd4edbafd"} Nov 04 00:58:46.661529 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.661432 5051 scope.go:122] "RemoveContainer" containerID="4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e" Nov 04 00:58:46.669445 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.669428 5051 scope.go:122] "RemoveContainer" containerID="4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e" Nov 04 00:58:46.669705 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:58:46.669687 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e\": container with ID starting with 4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e not found: ID does not exist" containerID="4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e" Nov 04 00:58:46.669761 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.669714 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e"} err="failed to get container status \"4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e\": rpc error: code = NotFound desc = could not find container \"4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e\": container with ID starting with 4880a8a35261a8cc6f3e15c59fdad10ea4f77b84c8d8bc37b0bf090ede22f89e not found: ID does not exist" Nov 04 00:58:46.719571 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.719552 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-config\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-config\") pod \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " Nov 04 00:58:46.719660 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.719594 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-tmp\") pod \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " Nov 04 00:58:46.719660 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.719626 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-trusted-ca-bundle\") pod \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " Nov 04 00:58:46.719660 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.719655 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-oauth-serving-cert\") pod \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " Nov 04 00:58:46.719808 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.719675 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-oauth-config\") pod \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " Nov 04 00:58:46.719808 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.719708 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-serving-cert\") pod \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " Nov 04 00:58:46.719808 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.719751 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-kube-api-access-flp79\" (UniqueName: \"kubernetes.io/projected/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-kube-api-access-flp79\") pod \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " Nov 04 00:58:46.719808 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.719795 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-service-ca\") pod \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\" (UID: \"1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8\") " Nov 04 00:58:46.720179 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.720094 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-tmp" pod "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" (UID: "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:58:46.720179 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.720027 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-config" pod "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" (UID: "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:46.720390 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.720342 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-oauth-serving-cert" pod "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" (UID: "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:46.720390 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.720356 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-trusted-ca-bundle" pod "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" (UID: "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:46.720492 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.720403 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-service-ca" pod "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" (UID: "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:58:46.722009 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.721983 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-kube-api-access-flp79" pod "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" (UID: "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8"). InnerVolumeSpecName "kube-api-access-flp79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:58:46.722306 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.722283 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-oauth-config" pod "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" (UID: "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:46.722402 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.722379 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-serving-cert" pod "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" (UID: "1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 00:58:46.820469 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.820412 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flp79\" (UniqueName: \"kubernetes.io/projected/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-kube-api-access-flp79\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:46.820469 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.820438 5051 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:46.820469 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.820454 5051 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-config\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:46.820469 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.820468 5051 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-tmp\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:46.820673 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.820482 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-trusted-ca-bundle\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:46.820673 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.820494 5051 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-oauth-serving-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:46.820673 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.820507 5051 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-oauth-config\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:46.820673 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.820520 5051 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8-console-serving-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:58:46.981066 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.981040 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6578864c88-mqrw8"] Nov 04 00:58:46.985327 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:46.985306 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6578864c88-mqrw8"] Nov 04 00:58:47.186278 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:58:47.186215 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" path="/var/lib/kubelet/pods/1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8/volumes" Nov 04 00:59:04.104597 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:59:04.104563 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 00:59:20.808176 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:20.808142 5051 generic.go:356] "Generic (PLEG): container finished" podID="552c2135-598a-4a6a-a71c-0d3fac0ecae4" containerID="4d441ec047106ce5a4fc19d666c636966b0b209e69e24d4efbde9a668b6da886" exitCode=0 Nov 04 00:59:20.808176 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:20.808154 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" event={"ID":"552c2135-598a-4a6a-a71c-0d3fac0ecae4","Type":"ContainerDied","Data":"4d441ec047106ce5a4fc19d666c636966b0b209e69e24d4efbde9a668b6da886"} Nov 04 00:59:20.810064 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:20.810041 5051 generic.go:356] "Generic (PLEG): container finished" podID="ede327f8-e1cf-4596-abe5-de379f528b15" containerID="307095b96a3aaff66e734627f934e11ddc0fc80f5b6d40da63ea7ae5be5fd243" exitCode=0 Nov 04 00:59:20.810191 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:20.810113 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" event={"ID":"ede327f8-e1cf-4596-abe5-de379f528b15","Type":"ContainerDied","Data":"307095b96a3aaff66e734627f934e11ddc0fc80f5b6d40da63ea7ae5be5fd243"} Nov 04 00:59:20.810191 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:20.810188 5051 scope.go:122] "RemoveContainer" containerID="00e74c9e1ed136bcdb85b5aef14228d27ea9e91ab2058c6cf4dc6df1668155c1" Nov 04 00:59:20.810505 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:20.810489 5051 scope.go:122] "RemoveContainer" containerID="307095b96a3aaff66e734627f934e11ddc0fc80f5b6d40da63ea7ae5be5fd243" Nov 04 00:59:20.810713 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:59:20.810691 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-sgnvg_openshift-insights(ede327f8-e1cf-4596-abe5-de379f528b15)\"" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" podUID="ede327f8-e1cf-4596-abe5-de379f528b15" Nov 04 00:59:21.973180 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:21.973154 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:59:22.096728 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.096651 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552c2135-598a-4a6a-a71c-0d3fac0ecae4-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552c2135-598a-4a6a-a71c-0d3fac0ecae4-archives-path\") pod \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " Nov 04 00:59:22.096728 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.096724 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-service-ca-bundle\") pod \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " Nov 04 00:59:22.096918 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.096743 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-trusted-ca-bundle\") pod \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " Nov 04 00:59:22.096918 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.096784 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552c2135-598a-4a6a-a71c-0d3fac0ecae4-kube-api-access-kn898\" (UniqueName: \"kubernetes.io/projected/552c2135-598a-4a6a-a71c-0d3fac0ecae4-kube-api-access-kn898\") pod \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\" (UID: \"552c2135-598a-4a6a-a71c-0d3fac0ecae4\") " Nov 04 00:59:22.097097 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.097002 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552c2135-598a-4a6a-a71c-0d3fac0ecae4-archives-path" pod "552c2135-598a-4a6a-a71c-0d3fac0ecae4" (UID: "552c2135-598a-4a6a-a71c-0d3fac0ecae4"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 00:59:22.097189 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.097079 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-service-ca-bundle" pod "552c2135-598a-4a6a-a71c-0d3fac0ecae4" (UID: "552c2135-598a-4a6a-a71c-0d3fac0ecae4"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:22.097189 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.097175 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-trusted-ca-bundle" pod "552c2135-598a-4a6a-a71c-0d3fac0ecae4" (UID: "552c2135-598a-4a6a-a71c-0d3fac0ecae4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 00:59:22.098756 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.098733 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552c2135-598a-4a6a-a71c-0d3fac0ecae4-kube-api-access-kn898" pod "552c2135-598a-4a6a-a71c-0d3fac0ecae4" (UID: "552c2135-598a-4a6a-a71c-0d3fac0ecae4"). InnerVolumeSpecName "kube-api-access-kn898". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 00:59:22.197715 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.197687 5051 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-service-ca-bundle\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:22.197715 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.197710 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/552c2135-598a-4a6a-a71c-0d3fac0ecae4-trusted-ca-bundle\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:22.197888 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.197723 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn898\" (UniqueName: \"kubernetes.io/projected/552c2135-598a-4a6a-a71c-0d3fac0ecae4-kube-api-access-kn898\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:22.197888 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.197738 5051 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/552c2135-598a-4a6a-a71c-0d3fac0ecae4-archives-path\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 00:59:22.821059 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.821026 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" event={"ID":"552c2135-598a-4a6a-a71c-0d3fac0ecae4","Type":"ContainerDied","Data":"522ac284fb5b92bc9da333359c8e175332c72e573abca4f8c6c10daf0f1901ef"} Nov 04 00:59:22.821059 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.821056 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-czn8m-jxhgx" Nov 04 00:59:22.821270 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:22.821072 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="522ac284fb5b92bc9da333359c8e175332c72e573abca4f8c6c10daf0f1901ef" Nov 04 00:59:34.104933 ip-10-0-137-48 kubenswrapper[5051]: E1104 00:59:34.104899 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 00:59:34.180822 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:34.180787 5051 scope.go:122] "RemoveContainer" containerID="307095b96a3aaff66e734627f934e11ddc0fc80f5b6d40da63ea7ae5be5fd243" Nov 04 00:59:34.872260 ip-10-0-137-48 kubenswrapper[5051]: I1104 00:59:34.872221 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-sgnvg" event={"ID":"ede327f8-e1cf-4596-abe5-de379f528b15","Type":"ContainerStarted","Data":"292173d844cf3a9fe42cb16e2aac0d4babf4aa75b84391b2bf4e04efbeec8b26"} Nov 04 01:00:04.104165 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:00:04.104076 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:00:34.104136 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:00:34.104079 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:01:04.104079 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:01:04.104043 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:01:34.104587 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:01:34.104547 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:02:04.103974 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:02:04.103937 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:02:09.686603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.686563 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7"] Nov 04 01:02:09.687072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687055 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3603f9b4-53ae-4e45-832f-e5c29ca743b9" containerName="registry" Nov 04 01:02:09.687116 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687076 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3603f9b4-53ae-4e45-832f-e5c29ca743b9" containerName="registry" Nov 04 01:02:09.687116 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687091 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c9027aeb-9f01-4ef8-b936-15c9382b2af8" containerName="insights-gathering" Nov 04 01:02:09.687116 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687099 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9027aeb-9f01-4ef8-b936-15c9382b2af8" containerName="insights-gathering" Nov 04 01:02:09.687240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687113 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cb337f11-c1da-42c0-9b61-7d47d41113ed" containerName="console" Nov 04 01:02:09.687240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687135 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb337f11-c1da-42c0-9b61-7d47d41113ed" containerName="console" Nov 04 01:02:09.687240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687155 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" containerName="console" Nov 04 01:02:09.687240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687163 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" containerName="console" Nov 04 01:02:09.687240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687176 5051 cpu_manager.go:414] "containerMap: removing container" podUID="552c2135-598a-4a6a-a71c-0d3fac0ecae4" containerName="insights-gathering" Nov 04 01:02:09.687240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687184 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552c2135-598a-4a6a-a71c-0d3fac0ecae4" containerName="insights-gathering" Nov 04 01:02:09.687240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687194 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e54204af-fcd5-442e-9009-f58ba803f3c4" containerName="registry" Nov 04 01:02:09.687240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687201 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54204af-fcd5-442e-9009-f58ba803f3c4" containerName="registry" Nov 04 01:02:09.687582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687304 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3603f9b4-53ae-4e45-832f-e5c29ca743b9" containerName="registry" Nov 04 01:02:09.687582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687316 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9027aeb-9f01-4ef8-b936-15c9382b2af8" containerName="insights-gathering" Nov 04 01:02:09.687582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687325 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb337f11-c1da-42c0-9b61-7d47d41113ed" containerName="console" Nov 04 01:02:09.687582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687335 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8dbde1-4579-4f9a-9e64-8fa5ad980ec8" containerName="console" Nov 04 01:02:09.687582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687343 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="552c2135-598a-4a6a-a71c-0d3fac0ecae4" containerName="insights-gathering" Nov 04 01:02:09.687582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.687351 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54204af-fcd5-442e-9009-f58ba803f3c4" containerName="registry" Nov 04 01:02:09.689485 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.689455 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.693281 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.693257 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 01:02:09.701929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.701899 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7"] Nov 04 01:02:09.727256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.727229 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f28wr\"" Nov 04 01:02:09.789432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789390 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.789603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789446 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01677e74-8528-41df-aa1a-947aefe19645-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.789603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789544 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.789723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789600 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.789723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789628 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01677e74-8528-41df-aa1a-947aefe19645-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.789809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789724 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.789809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789768 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.789907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789816 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gww\" (UniqueName: \"kubernetes.io/projected/01677e74-8528-41df-aa1a-947aefe19645-kube-api-access-j2gww\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.789907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.789844 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01677e74-8528-41df-aa1a-947aefe19645-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.890962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.890918 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gww\" (UniqueName: \"kubernetes.io/projected/01677e74-8528-41df-aa1a-947aefe19645-kube-api-access-j2gww\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.890962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.890964 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01677e74-8528-41df-aa1a-947aefe19645-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891096 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891163 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01677e74-8528-41df-aa1a-947aefe19645-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891202 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891243 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891265 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01677e74-8528-41df-aa1a-947aefe19645-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891304 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891354 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891599 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891672 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891707 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.891898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891876 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.892000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.891980 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01677e74-8528-41df-aa1a-947aefe19645-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.893589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.893567 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01677e74-8528-41df-aa1a-947aefe19645-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.893860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.893841 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01677e74-8528-41df-aa1a-947aefe19645-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.898400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.898381 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gww\" (UniqueName: \"kubernetes.io/projected/01677e74-8528-41df-aa1a-947aefe19645-kube-api-access-j2gww\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:09.898503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:09.898424 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01677e74-8528-41df-aa1a-947aefe19645-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xfhf7\" (UID: \"01677e74-8528-41df-aa1a-947aefe19645\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:10.007591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:10.007558 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:10.279692 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:10.279609 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7"] Nov 04 01:02:10.282647 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:02:10.282613 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01677e74_8528_41df_aa1a_947aefe19645.slice/crio-ec6f695407a9143cd6979c7a7e5eaa1f44605afe55eb60d46026e8de2a4ecc2d WatchSource:0}: Error finding container ec6f695407a9143cd6979c7a7e5eaa1f44605afe55eb60d46026e8de2a4ecc2d: Status 404 returned error can't find the container with id ec6f695407a9143cd6979c7a7e5eaa1f44605afe55eb60d46026e8de2a4ecc2d Nov 04 01:02:10.483333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:10.483298 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" event={"ID":"01677e74-8528-41df-aa1a-947aefe19645","Type":"ContainerStarted","Data":"ec6f695407a9143cd6979c7a7e5eaa1f44605afe55eb60d46026e8de2a4ecc2d"} Nov 04 01:02:19.089716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:19.089681 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:02:19.089976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:19.089757 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:02:19.089976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:19.089785 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:02:19.527010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:19.526976 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" event={"ID":"01677e74-8528-41df-aa1a-947aefe19645","Type":"ContainerStarted","Data":"cb12861cb6d47e136473fc21938e3e8fd12e47b0109a392ad0de11bd50e47305"} Nov 04 01:02:19.548514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:19.548466 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podStartSLOduration=1.744265725 podStartE2EDuration="10.548452685s" totalImagesPullingTime="8.80418696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:19.547253048 +0000 UTC m=+278.519001932" watchObservedRunningTime="2026-11-04 01:02:19.548452685 +0000 UTC m=+278.520201547" Nov 04 01:02:20.008309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:20.008277 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:20.013035 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:20.013008 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:20.531230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:20.531197 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:20.532305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:20.532285 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:02:34.103491 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:02:34.103450 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:02:41.195135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:41.195091 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:02:41.197395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:41.197370 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:02:53.945074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:53.944991 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk"] Nov 04 01:02:53.948414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:53.948397 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" Nov 04 01:02:53.955982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:53.955964 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 04 01:02:53.956064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:53.956010 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:02:53.956309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:53.956294 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-bk7zx\"" Nov 04 01:02:53.964976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:53.964949 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk"] Nov 04 01:02:54.066146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.066084 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddwt\" (UniqueName: \"kubernetes.io/projected/8200a967-2a54-4453-910d-ff48c7a55624-kube-api-access-dddwt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk\" (UID: \"8200a967-2a54-4453-910d-ff48c7a55624\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" Nov 04 01:02:54.066318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.066240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8200a967-2a54-4453-910d-ff48c7a55624-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk\" (UID: \"8200a967-2a54-4453-910d-ff48c7a55624\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" Nov 04 01:02:54.167248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.167206 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8200a967-2a54-4453-910d-ff48c7a55624-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk\" (UID: \"8200a967-2a54-4453-910d-ff48c7a55624\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" Nov 04 01:02:54.167420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.167263 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddwt\" (UniqueName: \"kubernetes.io/projected/8200a967-2a54-4453-910d-ff48c7a55624-kube-api-access-dddwt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk\" (UID: \"8200a967-2a54-4453-910d-ff48c7a55624\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" Nov 04 01:02:54.167625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.167604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8200a967-2a54-4453-910d-ff48c7a55624-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk\" (UID: \"8200a967-2a54-4453-910d-ff48c7a55624\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" Nov 04 01:02:54.182287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.182255 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddwt\" (UniqueName: \"kubernetes.io/projected/8200a967-2a54-4453-910d-ff48c7a55624-kube-api-access-dddwt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk\" (UID: \"8200a967-2a54-4453-910d-ff48c7a55624\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" Nov 04 01:02:54.260268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.260234 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" Nov 04 01:02:54.539015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.538986 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk"] Nov 04 01:02:54.542158 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:02:54.542099 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8200a967_2a54_4453_910d_ff48c7a55624.slice/crio-762365dc4bb96427e13851ac15b10e9febbfb7bb2f4d58f8b1632f71b8fd6cb8 WatchSource:0}: Error finding container 762365dc4bb96427e13851ac15b10e9febbfb7bb2f4d58f8b1632f71b8fd6cb8: Status 404 returned error can't find the container with id 762365dc4bb96427e13851ac15b10e9febbfb7bb2f4d58f8b1632f71b8fd6cb8 Nov 04 01:02:54.544931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.544912 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:02:54.673491 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:54.673455 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" event={"ID":"8200a967-2a54-4453-910d-ff48c7a55624","Type":"ContainerStarted","Data":"762365dc4bb96427e13851ac15b10e9febbfb7bb2f4d58f8b1632f71b8fd6cb8"} Nov 04 01:02:58.067905 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.067868 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk"] Nov 04 01:02:58.071980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.071957 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.103706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.103650 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk"] Nov 04 01:02:58.154182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.154146 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 01:02:58.154502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.154307 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 01:02:58.154502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.154396 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 01:02:58.154502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.154459 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 01:02:58.154649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.154563 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 01:02:58.154649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.154593 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9rtlr\"" Nov 04 01:02:58.204444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.204403 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59ac90c2-5b91-45fb-a1b2-b443b86668d8-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.204638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.204480 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.204638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.204603 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ac90c2-5b91-45fb-a1b2-b443b86668d8-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.204759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.204643 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.204759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.204663 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66qp\" (UniqueName: \"kubernetes.io/projected/59ac90c2-5b91-45fb-a1b2-b443b86668d8-kube-api-access-q66qp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.204866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.204762 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.305930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.305905 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ac90c2-5b91-45fb-a1b2-b443b86668d8-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.306043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.305940 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.306043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.305959 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66qp\" (UniqueName: \"kubernetes.io/projected/59ac90c2-5b91-45fb-a1b2-b443b86668d8-kube-api-access-q66qp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.306043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.305995 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.306043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.306026 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59ac90c2-5b91-45fb-a1b2-b443b86668d8-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.306299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.306077 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.306685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.306427 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59ac90c2-5b91-45fb-a1b2-b443b86668d8-manifests\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.308726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.308666 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.308813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.308780 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ac90c2-5b91-45fb-a1b2-b443b86668d8-tmp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.308909 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.308889 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.308984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.308956 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/59ac90c2-5b91-45fb-a1b2-b443b86668d8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.314958 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.314935 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66qp\" (UniqueName: \"kubernetes.io/projected/59ac90c2-5b91-45fb-a1b2-b443b86668d8-kube-api-access-q66qp\") pod \"opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk\" (UID: \"59ac90c2-5b91-45fb-a1b2-b443b86668d8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.390759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.390722 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:02:58.693974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.693879 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" event={"ID":"8200a967-2a54-4453-910d-ff48c7a55624","Type":"ContainerStarted","Data":"eefe6bff2773088473104fe29f2f9cfb439eead14158e926960a917184c2d62c"} Nov 04 01:02:58.703135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.703096 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk"] Nov 04 01:02:58.705734 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:02:58.705703 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ac90c2_5b91_45fb_a1b2_b443b86668d8.slice/crio-3bb2578b3f852482d4d9ec647ea78d46569e52ad6cdd2a3b6151626adc5690a7 WatchSource:0}: Error finding container 3bb2578b3f852482d4d9ec647ea78d46569e52ad6cdd2a3b6151626adc5690a7: Status 404 returned error can't find the container with id 3bb2578b3f852482d4d9ec647ea78d46569e52ad6cdd2a3b6151626adc5690a7 Nov 04 01:02:58.722583 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:58.722520 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-pb4mk" podStartSLOduration=1.971227866 podStartE2EDuration="5.722507292s" totalImagesPullingTime="3.751279426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:02:58.720908502 +0000 UTC m=+317.692657372" watchObservedRunningTime="2026-11-04 01:02:58.722507292 +0000 UTC m=+317.694256162" Nov 04 01:02:59.699727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:02:59.699679 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" event={"ID":"59ac90c2-5b91-45fb-a1b2-b443b86668d8","Type":"ContainerStarted","Data":"3bb2578b3f852482d4d9ec647ea78d46569e52ad6cdd2a3b6151626adc5690a7"} Nov 04 01:03:04.099841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.099763 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg"] Nov 04 01:03:04.104254 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:03:04.104223 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:03:04.106303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.106289 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:03:04.118408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.118385 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 01:03:04.118527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.118455 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 01:03:04.119557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.119539 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 04 01:03:04.119635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.119544 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lz954\"" Nov 04 01:03:04.131737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.131714 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg"] Nov 04 01:03:04.262664 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.262628 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k542n\" (UniqueName: \"kubernetes.io/projected/c9f5857b-b55b-4002-b4a3-4f61950bd6ba-kube-api-access-k542n\") pod \"dns-operator-controller-manager-8557cbdb84-tmmwg\" (UID: \"c9f5857b-b55b-4002-b4a3-4f61950bd6ba\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:03:04.363867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.363785 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k542n\" (UniqueName: \"kubernetes.io/projected/c9f5857b-b55b-4002-b4a3-4f61950bd6ba-kube-api-access-k542n\") pod \"dns-operator-controller-manager-8557cbdb84-tmmwg\" (UID: \"c9f5857b-b55b-4002-b4a3-4f61950bd6ba\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:03:04.375007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.374975 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k542n\" (UniqueName: \"kubernetes.io/projected/c9f5857b-b55b-4002-b4a3-4f61950bd6ba-kube-api-access-k542n\") pod \"dns-operator-controller-manager-8557cbdb84-tmmwg\" (UID: \"c9f5857b-b55b-4002-b4a3-4f61950bd6ba\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:03:04.418612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.418576 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:03:04.692014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.691789 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg"] Nov 04 01:03:04.694722 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:03:04.694698 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f5857b_b55b_4002_b4a3_4f61950bd6ba.slice/crio-7cb6b5451de1994c634e7793a8e97af6d99f0e3504d31a2661647fdfdbab0848 WatchSource:0}: Error finding container 7cb6b5451de1994c634e7793a8e97af6d99f0e3504d31a2661647fdfdbab0848: Status 404 returned error can't find the container with id 7cb6b5451de1994c634e7793a8e97af6d99f0e3504d31a2661647fdfdbab0848 Nov 04 01:03:04.722642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.722607 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" event={"ID":"c9f5857b-b55b-4002-b4a3-4f61950bd6ba","Type":"ContainerStarted","Data":"7cb6b5451de1994c634e7793a8e97af6d99f0e3504d31a2661647fdfdbab0848"} Nov 04 01:03:04.724405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.724372 5051 generic.go:356] "Generic (PLEG): container finished" podID="59ac90c2-5b91-45fb-a1b2-b443b86668d8" containerID="035f796191490993717bb250acff3ad6b7352821d6412712d34ec7ca9efb51e8" exitCode=0 Nov 04 01:03:04.724527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:04.724461 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" event={"ID":"59ac90c2-5b91-45fb-a1b2-b443b86668d8","Type":"ContainerDied","Data":"035f796191490993717bb250acff3ad6b7352821d6412712d34ec7ca9efb51e8"} Nov 04 01:03:05.730393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:05.730357 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" event={"ID":"59ac90c2-5b91-45fb-a1b2-b443b86668d8","Type":"ContainerStarted","Data":"7aa48ae41e39f9bc8beb9bb39b92a9bdaa259a5171443691286b530dade34560"} Nov 04 01:03:05.730806 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:05.730518 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:03:05.773072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:05.773022 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" podStartSLOduration=1.775748526 podStartE2EDuration="7.773009138s" totalImagesPullingTime="5.578577595s" totalInitContainerRuntime="418.683017ms" isStatefulPod=false podCreationTimestamp="2026-11-04 01:02:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:05.771641309 +0000 UTC m=+324.743390181" watchObservedRunningTime="2026-11-04 01:03:05.773009138 +0000 UTC m=+324.744758003" Nov 04 01:03:08.745585 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:08.745549 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" event={"ID":"c9f5857b-b55b-4002-b4a3-4f61950bd6ba","Type":"ContainerStarted","Data":"85b554c54591a77482b2b1e104692b9fac16a5fe0b6f6bf36c1043c5a770f690"} Nov 04 01:03:08.745971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:08.745663 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:03:08.774674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:08.774606 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" podStartSLOduration=1.002205852 podStartE2EDuration="4.774589955s" totalImagesPullingTime="3.772384103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:08.773051767 +0000 UTC m=+327.744800638" watchObservedRunningTime="2026-11-04 01:03:08.774589955 +0000 UTC m=+327.746338825" Nov 04 01:03:09.489610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.489574 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-zj4bp"] Nov 04 01:03:09.496389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.496365 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-zj4bp" Nov 04 01:03:09.501937 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.501909 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-zj4bp"] Nov 04 01:03:09.504199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.504180 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-m6bx6\"" Nov 04 01:03:09.504815 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.504785 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 01:03:09.504923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.504859 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 01:03:09.610172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.610109 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zdk\" (UniqueName: \"kubernetes.io/projected/fd2529a1-3127-46d9-ba8d-f1f60896abcc-kube-api-access-m7zdk\") pod \"cert-manager-597b95cb84-zj4bp\" (UID: \"fd2529a1-3127-46d9-ba8d-f1f60896abcc\") " pod="cert-manager/cert-manager-597b95cb84-zj4bp" Nov 04 01:03:09.610172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.610175 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd2529a1-3127-46d9-ba8d-f1f60896abcc-bound-sa-token\") pod \"cert-manager-597b95cb84-zj4bp\" (UID: \"fd2529a1-3127-46d9-ba8d-f1f60896abcc\") " pod="cert-manager/cert-manager-597b95cb84-zj4bp" Nov 04 01:03:09.711098 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.711059 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd2529a1-3127-46d9-ba8d-f1f60896abcc-bound-sa-token\") pod \"cert-manager-597b95cb84-zj4bp\" (UID: \"fd2529a1-3127-46d9-ba8d-f1f60896abcc\") " pod="cert-manager/cert-manager-597b95cb84-zj4bp" Nov 04 01:03:09.711319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.711210 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zdk\" (UniqueName: \"kubernetes.io/projected/fd2529a1-3127-46d9-ba8d-f1f60896abcc-kube-api-access-m7zdk\") pod \"cert-manager-597b95cb84-zj4bp\" (UID: \"fd2529a1-3127-46d9-ba8d-f1f60896abcc\") " pod="cert-manager/cert-manager-597b95cb84-zj4bp" Nov 04 01:03:09.720307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.720282 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd2529a1-3127-46d9-ba8d-f1f60896abcc-bound-sa-token\") pod \"cert-manager-597b95cb84-zj4bp\" (UID: \"fd2529a1-3127-46d9-ba8d-f1f60896abcc\") " pod="cert-manager/cert-manager-597b95cb84-zj4bp" Nov 04 01:03:09.720403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.720373 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zdk\" (UniqueName: \"kubernetes.io/projected/fd2529a1-3127-46d9-ba8d-f1f60896abcc-kube-api-access-m7zdk\") pod \"cert-manager-597b95cb84-zj4bp\" (UID: \"fd2529a1-3127-46d9-ba8d-f1f60896abcc\") " pod="cert-manager/cert-manager-597b95cb84-zj4bp" Nov 04 01:03:09.819940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:09.819866 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-zj4bp" Nov 04 01:03:10.095942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:10.095913 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-zj4bp"] Nov 04 01:03:10.098523 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:03:10.098496 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd2529a1_3127_46d9_ba8d_f1f60896abcc.slice/crio-64cd4d2b40ed01165bbc764280cded959b2876f803a028220c4d90276307eacb WatchSource:0}: Error finding container 64cd4d2b40ed01165bbc764280cded959b2876f803a028220c4d90276307eacb: Status 404 returned error can't find the container with id 64cd4d2b40ed01165bbc764280cded959b2876f803a028220c4d90276307eacb Nov 04 01:03:10.754718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:10.754679 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-zj4bp" event={"ID":"fd2529a1-3127-46d9-ba8d-f1f60896abcc","Type":"ContainerStarted","Data":"64cd4d2b40ed01165bbc764280cded959b2876f803a028220c4d90276307eacb"} Nov 04 01:03:14.776692 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:14.776661 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-zj4bp" event={"ID":"fd2529a1-3127-46d9-ba8d-f1f60896abcc","Type":"ContainerStarted","Data":"6692fa017af28a00ce4850affa590ffde7e1a8f42ea10f7effc2a40fed4432fa"} Nov 04 01:03:14.792091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:14.792050 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-zj4bp" podStartSLOduration=1.276521147 podStartE2EDuration="5.792038978s" totalImagesPullingTime="4.515517831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:14.792031946 +0000 UTC m=+333.763780816" watchObservedRunningTime="2026-11-04 01:03:14.792038978 +0000 UTC m=+333.763787848" Nov 04 01:03:15.871618 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:15.871583 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg"] Nov 04 01:03:15.874966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:15.874950 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:15.880529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:15.880508 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zklsh\"" Nov 04 01:03:15.886550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:15.886507 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg"] Nov 04 01:03:15.965650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:15.965622 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2493743-388a-4e6e-842f-e269caa53074-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtffg\" (UID: \"c2493743-388a-4e6e-842f-e269caa53074\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:15.965812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:15.965661 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv25n\" (UniqueName: \"kubernetes.io/projected/c2493743-388a-4e6e-842f-e269caa53074-kube-api-access-gv25n\") pod \"kuadrant-operator-controller-manager-678d87766f-wtffg\" (UID: \"c2493743-388a-4e6e-842f-e269caa53074\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:16.066155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:16.066104 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2493743-388a-4e6e-842f-e269caa53074-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtffg\" (UID: \"c2493743-388a-4e6e-842f-e269caa53074\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:16.066304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:16.066169 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv25n\" (UniqueName: \"kubernetes.io/projected/c2493743-388a-4e6e-842f-e269caa53074-kube-api-access-gv25n\") pod \"kuadrant-operator-controller-manager-678d87766f-wtffg\" (UID: \"c2493743-388a-4e6e-842f-e269caa53074\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:16.066566 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:16.066547 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2493743-388a-4e6e-842f-e269caa53074-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wtffg\" (UID: \"c2493743-388a-4e6e-842f-e269caa53074\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:16.075031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:16.075011 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv25n\" (UniqueName: \"kubernetes.io/projected/c2493743-388a-4e6e-842f-e269caa53074-kube-api-access-gv25n\") pod \"kuadrant-operator-controller-manager-678d87766f-wtffg\" (UID: \"c2493743-388a-4e6e-842f-e269caa53074\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:16.187614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:16.187544 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:16.474809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:16.474730 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg"] Nov 04 01:03:16.477129 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:03:16.477083 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2493743_388a_4e6e_842f_e269caa53074.slice/crio-14dd09737d348613f49ca9485bee7db3c4fb4290b1685b8d592c679ac5d7f1cc WatchSource:0}: Error finding container 14dd09737d348613f49ca9485bee7db3c4fb4290b1685b8d592c679ac5d7f1cc: Status 404 returned error can't find the container with id 14dd09737d348613f49ca9485bee7db3c4fb4290b1685b8d592c679ac5d7f1cc Nov 04 01:03:16.737543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:16.737518 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" Nov 04 01:03:16.787321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:16.787291 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" event={"ID":"c2493743-388a-4e6e-842f-e269caa53074","Type":"ContainerStarted","Data":"14dd09737d348613f49ca9485bee7db3c4fb4290b1685b8d592c679ac5d7f1cc"} Nov 04 01:03:19.752261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:19.752231 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:03:24.829174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:24.829140 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" event={"ID":"c2493743-388a-4e6e-842f-e269caa53074","Type":"ContainerStarted","Data":"0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578"} Nov 04 01:03:24.829562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:24.829213 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:24.854715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:24.854665 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" podStartSLOduration=2.282871707 podStartE2EDuration="9.854649938s" totalImagesPullingTime="7.571778231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:24.852101398 +0000 UTC m=+343.823850269" watchObservedRunningTime="2026-11-04 01:03:24.854649938 +0000 UTC m=+343.826398812" Nov 04 01:03:32.256864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.256821 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6"] Nov 04 01:03:32.261768 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.261744 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.275161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.275109 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fk5pr\"" Nov 04 01:03:32.277691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.277660 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6"] Nov 04 01:03:32.313799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.313762 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.313974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.313808 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.313974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.313907 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.313974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.313937 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.313974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.313956 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5wkx\" (UniqueName: \"kubernetes.io/projected/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-kube-api-access-l5wkx\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.314107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.313986 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.314107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.314085 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.314187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.314106 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.314187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.314131 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.414816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.414784 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.414990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.414836 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.414990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.414951 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.414992 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415018 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415060 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415111 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415189 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415201 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415219 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5wkx\" (UniqueName: \"kubernetes.io/projected/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-kube-api-access-l5wkx\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415469 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415611 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415635 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.415812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.415612 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.417414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.417392 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.417670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.417648 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.426687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.426660 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.427167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.427149 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5wkx\" (UniqueName: \"kubernetes.io/projected/6d63414f-bab7-4ddc-b3fe-aa0da66e7881-kube-api-access-l5wkx\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6\" (UID: \"6d63414f-bab7-4ddc-b3fe-aa0da66e7881\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.578306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.578222 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:32.868708 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.862707 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6"] Nov 04 01:03:32.872049 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:03:32.872005 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d63414f_bab7_4ddc_b3fe_aa0da66e7881.slice/crio-7d7e1d110d4d6ba0b871bd3ae57f9deeea511eb91df5c470f3444c7801a238fe WatchSource:0}: Error finding container 7d7e1d110d4d6ba0b871bd3ae57f9deeea511eb91df5c470f3444c7801a238fe: Status 404 returned error can't find the container with id 7d7e1d110d4d6ba0b871bd3ae57f9deeea511eb91df5c470f3444c7801a238fe Nov 04 01:03:32.874012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.873975 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:03:32.874089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.874043 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:03:32.874089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:32.874070 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:03:33.846156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.846099 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn"] Nov 04 01:03:33.849337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.849314 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:33.852891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.852855 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 01:03:33.853024 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.852946 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 01:03:33.856526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.856500 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 04 01:03:33.861018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.860992 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn"] Nov 04 01:03:33.879327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.879299 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" event={"ID":"6d63414f-bab7-4ddc-b3fe-aa0da66e7881","Type":"ContainerStarted","Data":"e0ebfa9f44b4b794d3e1608d4e028cf0b1ea2d8e89277decc6b897844f2d0f80"} Nov 04 01:03:33.879476 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.879347 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" event={"ID":"6d63414f-bab7-4ddc-b3fe-aa0da66e7881","Type":"ContainerStarted","Data":"7d7e1d110d4d6ba0b871bd3ae57f9deeea511eb91df5c470f3444c7801a238fe"} Nov 04 01:03:33.899158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.899096 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" podStartSLOduration=1.8990835480000001 podStartE2EDuration="1.899083548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:33.898004689 +0000 UTC m=+352.869753559" watchObservedRunningTime="2026-11-04 01:03:33.899083548 +0000 UTC m=+352.870832418" Nov 04 01:03:33.909346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.909321 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-r2rvj\"" Nov 04 01:03:33.928465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.928431 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:33.928552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.928496 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:33.928696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.928674 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:33.928838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:33.928817 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6ts\" (UniqueName: \"kubernetes.io/projected/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-kube-api-access-zn6ts\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.030184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.030114 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6ts\" (UniqueName: \"kubernetes.io/projected/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-kube-api-access-zn6ts\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.030382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.030206 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.030382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.030228 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.030382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.030260 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.030701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.030670 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.030701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.030696 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.031242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.031221 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.038447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.038424 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6ts\" (UniqueName: \"kubernetes.io/projected/11a8492c-3577-4e3c-b7ab-a8f84b5d8b03-kube-api-access-zn6ts\") pod \"kserve-module-controller-manager-7566cf5fc8-ph6rn\" (UID: \"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.104429 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:03:34.104345 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:03:34.164195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.164144 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:34.440740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.440716 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn"] Nov 04 01:03:34.443436 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:03:34.443411 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a8492c_3577_4e3c_b7ab_a8f84b5d8b03.slice/crio-58cbcecfd3fd832d61d14a52fa98e26a82b2db9fb5851cc072c806992de4cb74 WatchSource:0}: Error finding container 58cbcecfd3fd832d61d14a52fa98e26a82b2db9fb5851cc072c806992de4cb74: Status 404 returned error can't find the container with id 58cbcecfd3fd832d61d14a52fa98e26a82b2db9fb5851cc072c806992de4cb74 Nov 04 01:03:34.578573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.578541 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:34.583589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.583560 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:34.886045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.886013 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" event={"ID":"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03","Type":"ContainerStarted","Data":"58cbcecfd3fd832d61d14a52fa98e26a82b2db9fb5851cc072c806992de4cb74"} Nov 04 01:03:34.886453 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.886223 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:34.887073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:34.887058 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6" Nov 04 01:03:35.836320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:35.836284 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:03:38.909563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:38.909528 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" event={"ID":"11a8492c-3577-4e3c-b7ab-a8f84b5d8b03","Type":"ContainerStarted","Data":"575aaecc125200a67497cec3b5d69f62ed7cb21f6bd6a50e76ce8d832ef887ce"} Nov 04 01:03:38.909935 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:38.909600 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:38.929678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:38.929625 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" podStartSLOduration=2.393520502 podStartE2EDuration="5.929612896s" totalImagesPullingTime="3.536092394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:38.927307054 +0000 UTC m=+357.899055928" watchObservedRunningTime="2026-11-04 01:03:38.929612896 +0000 UTC m=+357.901361767" Nov 04 01:03:43.885884 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.885845 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv"] Nov 04 01:03:43.892463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.892441 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:43.900235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.900212 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 01:03:43.901452 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.901430 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 04 01:03:43.901452 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.901430 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-5x9xr\"" Nov 04 01:03:43.904441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.904424 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 04 01:03:43.912814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.912794 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv"] Nov 04 01:03:43.999317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.999294 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 04 01:03:43.999445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:43.999429 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 01:03:44.018930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.018904 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-cert\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.019042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.019026 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-manager-config\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.019102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.019062 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-metrics-certs\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.019209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.019187 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2drk\" (UniqueName: \"kubernetes.io/projected/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-kube-api-access-k2drk\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.120354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.120330 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-manager-config\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.120459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.120363 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-metrics-certs\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.120459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.120382 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2drk\" (UniqueName: \"kubernetes.io/projected/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-kube-api-access-k2drk\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.120459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.120419 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-cert\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.120955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.120937 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-manager-config\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.123159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.123139 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-cert\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.123215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.123148 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-metrics-certs\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.128077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.128056 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2drk\" (UniqueName: \"kubernetes.io/projected/9d5e6ac7-97c0-407d-b050-bf4d8670b7e9-kube-api-access-k2drk\") pod \"jobset-controller-manager-69d5854bb-87mwv\" (UID: \"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9\") " pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.205211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.205150 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:44.475012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.474960 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv"] Nov 04 01:03:44.477109 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:03:44.477085 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5e6ac7_97c0_407d_b050_bf4d8670b7e9.slice/crio-2e420195d36fb3650b2049439a82262e8f6b3ec9e1ad5be61e9f21818be648af WatchSource:0}: Error finding container 2e420195d36fb3650b2049439a82262e8f6b3ec9e1ad5be61e9f21818be648af: Status 404 returned error can't find the container with id 2e420195d36fb3650b2049439a82262e8f6b3ec9e1ad5be61e9f21818be648af Nov 04 01:03:44.937361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:44.937326 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" event={"ID":"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9","Type":"ContainerStarted","Data":"2e420195d36fb3650b2049439a82262e8f6b3ec9e1ad5be61e9f21818be648af"} Nov 04 01:03:48.960878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:48.960789 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" event={"ID":"9d5e6ac7-97c0-407d-b050-bf4d8670b7e9","Type":"ContainerStarted","Data":"07b4cac5b69d05bf2e217b99acdfc2c22f462690705b4fd76b25a6641ce6f70c"} Nov 04 01:03:48.960878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:48.960865 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:03:48.977731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:48.977681 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" podStartSLOduration=1.809811674 podStartE2EDuration="5.977667076s" totalImagesPullingTime="4.167855402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:03:48.976037746 +0000 UTC m=+367.947786642" watchObservedRunningTime="2026-11-04 01:03:48.977667076 +0000 UTC m=+367.949415949" Nov 04 01:03:59.915041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:59.915004 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ph6rn" Nov 04 01:03:59.970979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:03:59.970952 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:04:04.104715 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:04:04.104680 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:04:34.105767 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:04:34.105727 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:05:04.106203 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:05:04.106167 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:05:34.104167 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:05:34.104112 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:06:04.104496 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:06:04.104419 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:06:34.103233 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:06:34.103191 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:07:04.104237 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:07:04.104133 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:07:34.104441 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:07:34.104358 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:07:41.222540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:07:41.222505 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:07:41.225818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:07:41.225796 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:08:04.104302 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:08:04.104255 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:08:34.104190 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:08:34.104151 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:09:04.104352 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:09:04.104273 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:09:34.104733 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:09:34.104694 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:10:04.104191 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:10:04.104142 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:10:34.103607 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:10:34.103525 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:11:04.103780 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:11:04.103741 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:11:34.104966 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:11:34.104927 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:12:04.104184 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:12:04.104149 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:12:34.104109 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:12:34.104070 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:12:41.251377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:12:41.251352 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:12:41.255506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:12:41.255486 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:13:04.104496 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:13:04.104458 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:13:34.105354 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:13:34.105266 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:13:52.154802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.154769 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-wtffg_c2493743-388a-4e6e-842f-e269caa53074/manager/0.log" Nov 04 01:13:52.352321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.352289 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg"] Nov 04 01:13:52.352508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.352484 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" podUID="c2493743-388a-4e6e-842f-e269caa53074" containerName="manager" containerID="cri-o://0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578" gracePeriod=10 Nov 04 01:13:52.376732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.376705 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w"] Nov 04 01:13:52.379959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.379942 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:52.393335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.393304 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w"] Nov 04 01:13:52.428897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.428834 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gmc9w\" (UID: \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:52.428897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.428864 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4rj\" (UniqueName: \"kubernetes.io/projected/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-kube-api-access-8t4rj\") pod \"kuadrant-operator-controller-manager-678d87766f-gmc9w\" (UID: \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:52.529822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.529781 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gmc9w\" (UID: \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:52.529957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.529833 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4rj\" (UniqueName: \"kubernetes.io/projected/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-kube-api-access-8t4rj\") pod \"kuadrant-operator-controller-manager-678d87766f-gmc9w\" (UID: \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:52.530249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.530226 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gmc9w\" (UID: \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:52.539303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.539281 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4rj\" (UniqueName: \"kubernetes.io/projected/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-kube-api-access-8t4rj\") pod \"kuadrant-operator-controller-manager-678d87766f-gmc9w\" (UID: \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:52.608254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.608232 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:13:52.703043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.702966 5051 generic.go:356] "Generic (PLEG): container finished" podID="c2493743-388a-4e6e-842f-e269caa53074" containerID="0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578" exitCode=0 Nov 04 01:13:52.703043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.703014 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" Nov 04 01:13:52.703043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.703022 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" event={"ID":"c2493743-388a-4e6e-842f-e269caa53074","Type":"ContainerDied","Data":"0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578"} Nov 04 01:13:52.703289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.703063 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg" event={"ID":"c2493743-388a-4e6e-842f-e269caa53074","Type":"ContainerDied","Data":"14dd09737d348613f49ca9485bee7db3c4fb4290b1685b8d592c679ac5d7f1cc"} Nov 04 01:13:52.703289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.703084 5051 scope.go:122] "RemoveContainer" containerID="0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578" Nov 04 01:13:52.711344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.711327 5051 scope.go:122] "RemoveContainer" containerID="0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578" Nov 04 01:13:52.711613 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:13:52.711590 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578\": container with ID starting with 0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578 not found: ID does not exist" containerID="0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578" Nov 04 01:13:52.711673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.711624 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578"} err="failed to get container status \"0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578\": rpc error: code = NotFound desc = could not find container \"0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578\": container with ID starting with 0ad1e0db585cfd7dcb81cb5b96cacedd8ae532c8a524c2f559c9ca3585d1c578 not found: ID does not exist" Nov 04 01:13:52.730911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.730889 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2493743-388a-4e6e-842f-e269caa53074-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2493743-388a-4e6e-842f-e269caa53074-extensions-socket-volume\") pod \"c2493743-388a-4e6e-842f-e269caa53074\" (UID: \"c2493743-388a-4e6e-842f-e269caa53074\") " Nov 04 01:13:52.731012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.730967 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2493743-388a-4e6e-842f-e269caa53074-kube-api-access-gv25n\" (UniqueName: \"kubernetes.io/projected/c2493743-388a-4e6e-842f-e269caa53074-kube-api-access-gv25n\") pod \"c2493743-388a-4e6e-842f-e269caa53074\" (UID: \"c2493743-388a-4e6e-842f-e269caa53074\") " Nov 04 01:13:52.731246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.731224 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2493743-388a-4e6e-842f-e269caa53074-extensions-socket-volume" pod "c2493743-388a-4e6e-842f-e269caa53074" (UID: "c2493743-388a-4e6e-842f-e269caa53074"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:13:52.733003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.732981 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2493743-388a-4e6e-842f-e269caa53074-kube-api-access-gv25n" pod "c2493743-388a-4e6e-842f-e269caa53074" (UID: "c2493743-388a-4e6e-842f-e269caa53074"). InnerVolumeSpecName "kube-api-access-gv25n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:13:52.741253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.741230 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:52.831836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.831792 5051 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2493743-388a-4e6e-842f-e269caa53074-extensions-socket-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:13:52.831836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:52.831834 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv25n\" (UniqueName: \"kubernetes.io/projected/c2493743-388a-4e6e-842f-e269caa53074-kube-api-access-gv25n\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:13:53.019515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:53.019440 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w"] Nov 04 01:13:53.022028 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:13:53.022003 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c49724a_9d0f_42ec_86c8_cf8f7229f38a.slice/crio-284b2b317afcdbc62b18872bf850edc4a5d7c730a1bb360e25d450e10cfed495 WatchSource:0}: Error finding container 284b2b317afcdbc62b18872bf850edc4a5d7c730a1bb360e25d450e10cfed495: Status 404 returned error can't find the container with id 284b2b317afcdbc62b18872bf850edc4a5d7c730a1bb360e25d450e10cfed495 Nov 04 01:13:53.027433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:53.027412 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg"] Nov 04 01:13:53.033362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:53.031830 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wtffg"] Nov 04 01:13:53.186543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:53.186498 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2493743-388a-4e6e-842f-e269caa53074" path="/var/lib/kubelet/pods/c2493743-388a-4e6e-842f-e269caa53074/volumes" Nov 04 01:13:53.710547 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:53.710510 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" event={"ID":"5c49724a-9d0f-42ec-86c8-cf8f7229f38a","Type":"ContainerStarted","Data":"d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3"} Nov 04 01:13:53.710764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:53.710569 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" event={"ID":"5c49724a-9d0f-42ec-86c8-cf8f7229f38a","Type":"ContainerStarted","Data":"284b2b317afcdbc62b18872bf850edc4a5d7c730a1bb360e25d450e10cfed495"} Nov 04 01:13:53.710764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:53.710646 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:13:53.742562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:13:53.742512 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" podStartSLOduration=1.7424985689999999 podStartE2EDuration="1.742498569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:13:53.74096739 +0000 UTC m=+972.712716257" watchObservedRunningTime="2026-11-04 01:13:53.742498569 +0000 UTC m=+972.714247440" Nov 04 01:14:04.104637 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:14:04.104591 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:14:04.717424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:04.717391 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:14:09.572977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.572945 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bzqkz"] Nov 04 01:14:09.573470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.573352 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c2493743-388a-4e6e-842f-e269caa53074" containerName="manager" Nov 04 01:14:09.573470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.573365 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2493743-388a-4e6e-842f-e269caa53074" containerName="manager" Nov 04 01:14:09.573470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.573450 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2493743-388a-4e6e-842f-e269caa53074" containerName="manager" Nov 04 01:14:09.576457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.576437 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" Nov 04 01:14:09.583173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.583151 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bzqkz"] Nov 04 01:14:09.584328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.584307 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xvwz\"" Nov 04 01:14:09.681650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.681609 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cx87\" (UniqueName: \"kubernetes.io/projected/d8f97df2-bf5e-484f-98fb-2a6e8b2dc316-kube-api-access-4cx87\") pod \"authorino-56c6c9ddd5-bzqkz\" (UID: \"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316\") " pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" Nov 04 01:14:09.782167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.782137 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cx87\" (UniqueName: \"kubernetes.io/projected/d8f97df2-bf5e-484f-98fb-2a6e8b2dc316-kube-api-access-4cx87\") pod \"authorino-56c6c9ddd5-bzqkz\" (UID: \"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316\") " pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" Nov 04 01:14:09.790978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.790953 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cx87\" (UniqueName: \"kubernetes.io/projected/d8f97df2-bf5e-484f-98fb-2a6e8b2dc316-kube-api-access-4cx87\") pod \"authorino-56c6c9ddd5-bzqkz\" (UID: \"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316\") " pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" Nov 04 01:14:09.890492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:09.890421 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" Nov 04 01:14:10.159265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:10.159239 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bzqkz"] Nov 04 01:14:10.161591 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:14:10.161559 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8f97df2_bf5e_484f_98fb_2a6e8b2dc316.slice/crio-d043606af98f8b55a8a46d60e8781cb8eb989c4557f6a0eedcfc1fd0146e0a92 WatchSource:0}: Error finding container d043606af98f8b55a8a46d60e8781cb8eb989c4557f6a0eedcfc1fd0146e0a92: Status 404 returned error can't find the container with id d043606af98f8b55a8a46d60e8781cb8eb989c4557f6a0eedcfc1fd0146e0a92 Nov 04 01:14:10.163006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:10.162982 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:14:10.788137 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:10.788084 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" event={"ID":"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316","Type":"ContainerStarted","Data":"d043606af98f8b55a8a46d60e8781cb8eb989c4557f6a0eedcfc1fd0146e0a92"} Nov 04 01:14:12.799316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:14:12.799279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" event={"ID":"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316","Type":"ContainerStarted","Data":"fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe"} Nov 04 01:14:34.103746 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:14:34.103706 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:15:04.104533 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:15:04.104490 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:15:32.935320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:32.935268 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" podStartSLOduration=81.702967862 podStartE2EDuration="1m23.935252715s" totalImagesPullingTime="2.232284853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:14:12.812937671 +0000 UTC m=+991.784686547" watchObservedRunningTime="2026-11-04 01:15:32.935252715 +0000 UTC m=+1071.907001586" Nov 04 01:15:32.935757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:32.935670 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n8lt5"] Nov 04 01:15:32.939198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:32.939178 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:32.944561 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:32.944537 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 01:15:32.945870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:32.945845 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n8lt5"] Nov 04 01:15:33.011477 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.011451 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhx7\" (UniqueName: \"kubernetes.io/projected/50c57a68-c92e-4459-931b-a794781b3b0a-kube-api-access-fjhx7\") pod \"authorino-6889cbd8c4-n8lt5\" (UID: \"50c57a68-c92e-4459-931b-a794781b3b0a\") " pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:33.011639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.011567 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50c57a68-c92e-4459-931b-a794781b3b0a-tls-cert\") pod \"authorino-6889cbd8c4-n8lt5\" (UID: \"50c57a68-c92e-4459-931b-a794781b3b0a\") " pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:33.112638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.112600 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50c57a68-c92e-4459-931b-a794781b3b0a-tls-cert\") pod \"authorino-6889cbd8c4-n8lt5\" (UID: \"50c57a68-c92e-4459-931b-a794781b3b0a\") " pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:33.112792 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.112687 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhx7\" (UniqueName: \"kubernetes.io/projected/50c57a68-c92e-4459-931b-a794781b3b0a-kube-api-access-fjhx7\") pod \"authorino-6889cbd8c4-n8lt5\" (UID: \"50c57a68-c92e-4459-931b-a794781b3b0a\") " pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:33.115108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.115079 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50c57a68-c92e-4459-931b-a794781b3b0a-tls-cert\") pod \"authorino-6889cbd8c4-n8lt5\" (UID: \"50c57a68-c92e-4459-931b-a794781b3b0a\") " pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:33.121998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.121967 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhx7\" (UniqueName: \"kubernetes.io/projected/50c57a68-c92e-4459-931b-a794781b3b0a-kube-api-access-fjhx7\") pod \"authorino-6889cbd8c4-n8lt5\" (UID: \"50c57a68-c92e-4459-931b-a794781b3b0a\") " pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:33.170247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.170217 5051 generic.go:356] "Generic (PLEG): container finished" podID="01677e74-8528-41df-aa1a-947aefe19645" containerID="cb12861cb6d47e136473fc21938e3e8fd12e47b0109a392ad0de11bd50e47305" exitCode=0 Nov 04 01:15:33.170373 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.170293 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" event={"ID":"01677e74-8528-41df-aa1a-947aefe19645","Type":"ContainerDied","Data":"cb12861cb6d47e136473fc21938e3e8fd12e47b0109a392ad0de11bd50e47305"} Nov 04 01:15:33.170650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.170636 5051 scope.go:122] "RemoveContainer" containerID="cb12861cb6d47e136473fc21938e3e8fd12e47b0109a392ad0de11bd50e47305" Nov 04 01:15:33.171705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.171677 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:33.171767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.171752 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:33.171807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.171786 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:15:33.253179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.253150 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:33.269949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.269919 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n8lt5"] Nov 04 01:15:33.546246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:33.546157 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n8lt5"] Nov 04 01:15:34.105296 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:15:34.105258 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:15:34.179027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:34.178990 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" event={"ID":"50c57a68-c92e-4459-931b-a794781b3b0a","Type":"ContainerStarted","Data":"02bd3b32c9acdfa0cf2a7a67ac94bd207c9b37978378a9c8356006e6600229dd"} Nov 04 01:15:34.181184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:34.181159 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" event={"ID":"01677e74-8528-41df-aa1a-947aefe19645","Type":"ContainerStarted","Data":"0914e6ef5ce1ed101a420c49cab648314414126ed5b7dbbb824146aaec7f54d0"} Nov 04 01:15:35.012824 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.010936 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:15:35.016748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.014139 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:15:35.016748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.014219 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:15:35.193521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.193264 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" podUID="50c57a68-c92e-4459-931b-a794781b3b0a" containerName="authorino" containerID="cri-o://88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59" gracePeriod=30 Nov 04 01:15:35.195319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.195290 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" event={"ID":"50c57a68-c92e-4459-931b-a794781b3b0a","Type":"ContainerStarted","Data":"88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59"} Nov 04 01:15:35.208650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.208598 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" podStartSLOduration=2.34278738 podStartE2EDuration="3.208586273s" totalImagesPullingTime="865.798893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:35.20778232 +0000 UTC m=+1074.179531190" watchObservedRunningTime="2026-11-04 01:15:35.208586273 +0000 UTC m=+1074.180335144" Nov 04 01:15:35.280890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.280801 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bzqkz"] Nov 04 01:15:35.281434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.281026 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" podUID="d8f97df2-bf5e-484f-98fb-2a6e8b2dc316" containerName="authorino" containerID="cri-o://fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe" gracePeriod=30 Nov 04 01:15:35.482538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.482511 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:35.532096 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.532023 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:15:35.539376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.539350 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c57a68-c92e-4459-931b-a794781b3b0a-kube-api-access-fjhx7\" (UniqueName: \"kubernetes.io/projected/50c57a68-c92e-4459-931b-a794781b3b0a-kube-api-access-fjhx7\") pod \"50c57a68-c92e-4459-931b-a794781b3b0a\" (UID: \"50c57a68-c92e-4459-931b-a794781b3b0a\") " Nov 04 01:15:35.539479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.539395 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50c57a68-c92e-4459-931b-a794781b3b0a-tls-cert\" (UniqueName: \"kubernetes.io/secret/50c57a68-c92e-4459-931b-a794781b3b0a-tls-cert\") pod \"50c57a68-c92e-4459-931b-a794781b3b0a\" (UID: \"50c57a68-c92e-4459-931b-a794781b3b0a\") " Nov 04 01:15:35.542408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.542381 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c57a68-c92e-4459-931b-a794781b3b0a-kube-api-access-fjhx7" pod "50c57a68-c92e-4459-931b-a794781b3b0a" (UID: "50c57a68-c92e-4459-931b-a794781b3b0a"). InnerVolumeSpecName "kube-api-access-fjhx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:35.579957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.579922 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c57a68-c92e-4459-931b-a794781b3b0a-tls-cert" pod "50c57a68-c92e-4459-931b-a794781b3b0a" (UID: "50c57a68-c92e-4459-931b-a794781b3b0a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:15:35.585951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.585930 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" Nov 04 01:15:35.640633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.640599 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f97df2-bf5e-484f-98fb-2a6e8b2dc316-kube-api-access-4cx87\" (UniqueName: \"kubernetes.io/projected/d8f97df2-bf5e-484f-98fb-2a6e8b2dc316-kube-api-access-4cx87\") pod \"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316\" (UID: \"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316\") " Nov 04 01:15:35.640860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.640848 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjhx7\" (UniqueName: \"kubernetes.io/projected/50c57a68-c92e-4459-931b-a794781b3b0a-kube-api-access-fjhx7\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:35.640915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.640862 5051 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/50c57a68-c92e-4459-931b-a794781b3b0a-tls-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:35.642649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.642623 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f97df2-bf5e-484f-98fb-2a6e8b2dc316-kube-api-access-4cx87" pod "d8f97df2-bf5e-484f-98fb-2a6e8b2dc316" (UID: "d8f97df2-bf5e-484f-98fb-2a6e8b2dc316"). InnerVolumeSpecName "kube-api-access-4cx87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:15:35.742337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:35.742300 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cx87\" (UniqueName: \"kubernetes.io/projected/d8f97df2-bf5e-484f-98fb-2a6e8b2dc316-kube-api-access-4cx87\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:15:36.007954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.007923 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:15:36.008159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.008007 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:15:36.200936 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.200905 5051 generic.go:356] "Generic (PLEG): container finished" podID="50c57a68-c92e-4459-931b-a794781b3b0a" containerID="88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59" exitCode=0 Nov 04 01:15:36.201401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.200949 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" Nov 04 01:15:36.201401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.200991 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" event={"ID":"50c57a68-c92e-4459-931b-a794781b3b0a","Type":"ContainerDied","Data":"88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59"} Nov 04 01:15:36.201401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.201041 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-n8lt5" event={"ID":"50c57a68-c92e-4459-931b-a794781b3b0a","Type":"ContainerDied","Data":"02bd3b32c9acdfa0cf2a7a67ac94bd207c9b37978378a9c8356006e6600229dd"} Nov 04 01:15:36.201401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.201066 5051 scope.go:122] "RemoveContainer" containerID="88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59" Nov 04 01:15:36.202391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.202360 5051 generic.go:356] "Generic (PLEG): container finished" podID="d8f97df2-bf5e-484f-98fb-2a6e8b2dc316" containerID="fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe" exitCode=0 Nov 04 01:15:36.202489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.202405 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" Nov 04 01:15:36.202489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.202450 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" event={"ID":"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316","Type":"ContainerDied","Data":"fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe"} Nov 04 01:15:36.202602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.202492 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bzqkz" event={"ID":"d8f97df2-bf5e-484f-98fb-2a6e8b2dc316","Type":"ContainerDied","Data":"d043606af98f8b55a8a46d60e8781cb8eb989c4557f6a0eedcfc1fd0146e0a92"} Nov 04 01:15:36.223534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.223516 5051 scope.go:122] "RemoveContainer" containerID="88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59" Nov 04 01:15:36.223804 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:15:36.223783 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59\": container with ID starting with 88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59 not found: ID does not exist" containerID="88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59" Nov 04 01:15:36.223888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.223810 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59"} err="failed to get container status \"88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59\": rpc error: code = NotFound desc = could not find container \"88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59\": container with ID starting with 88d6f048aeef1ce15ddee0abf13a766dd939016ad764ae8def854c3fb434fa59 not found: ID does not exist" Nov 04 01:15:36.223888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.223826 5051 scope.go:122] "RemoveContainer" containerID="fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe" Nov 04 01:15:36.231669 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.231651 5051 scope.go:122] "RemoveContainer" containerID="fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe" Nov 04 01:15:36.231925 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:15:36.231907 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe\": container with ID starting with fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe not found: ID does not exist" containerID="fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe" Nov 04 01:15:36.231965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.231932 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe"} err="failed to get container status \"fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe\": rpc error: code = NotFound desc = could not find container \"fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe\": container with ID starting with fc052d398c7552e1ee3ab28c93e4ae67f8da3ba0fff758d201e4fa0c8ae73dfe not found: ID does not exist" Nov 04 01:15:36.235372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.235353 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bzqkz"] Nov 04 01:15:36.237570 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.237551 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bzqkz"] Nov 04 01:15:36.248108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.248078 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n8lt5"] Nov 04 01:15:36.273185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:36.272862 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-n8lt5"] Nov 04 01:15:37.008087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:37.008052 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:15:37.008288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:37.008145 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:15:37.187576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:37.187543 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c57a68-c92e-4459-931b-a794781b3b0a" path="/var/lib/kubelet/pods/50c57a68-c92e-4459-931b-a794781b3b0a/volumes" Nov 04 01:15:37.187891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:37.187876 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f97df2-bf5e-484f-98fb-2a6e8b2dc316" path="/var/lib/kubelet/pods/d8f97df2-bf5e-484f-98fb-2a6e8b2dc316/volumes" Nov 04 01:15:38.008374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:38.008340 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:15:38.008781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:38.008388 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:15:39.008132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:39.008087 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Nov 04 01:15:39.008315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:39.008171 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Nov 04 01:15:40.012149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:40.012110 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:15:40.013085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:40.013067 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" Nov 04 01:15:44.459203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.459163 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-cfwcb-f2q7x"] Nov 04 01:15:44.459775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.459755 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d8f97df2-bf5e-484f-98fb-2a6e8b2dc316" containerName="authorino" Nov 04 01:15:44.459830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.459778 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f97df2-bf5e-484f-98fb-2a6e8b2dc316" containerName="authorino" Nov 04 01:15:44.459830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.459792 5051 cpu_manager.go:414] "containerMap: removing container" podUID="50c57a68-c92e-4459-931b-a794781b3b0a" containerName="authorino" Nov 04 01:15:44.459830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.459800 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c57a68-c92e-4459-931b-a794781b3b0a" containerName="authorino" Nov 04 01:15:44.459923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.459889 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c57a68-c92e-4459-931b-a794781b3b0a" containerName="authorino" Nov 04 01:15:44.459923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.459913 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f97df2-bf5e-484f-98fb-2a6e8b2dc316" containerName="authorino" Nov 04 01:15:44.464788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.464769 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.468702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.468676 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cfwcb-f2q7x"] Nov 04 01:15:44.538194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.538156 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-trusted-ca-bundle\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.538194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.538198 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9f6z\" (UniqueName: \"kubernetes.io/projected/608e1cfd-3e3f-4bad-b275-74805261379b-kube-api-access-g9f6z\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.538386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.538221 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/608e1cfd-3e3f-4bad-b275-74805261379b-archives-path\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.538386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.538354 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-service-ca-bundle\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.639521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.639482 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-trusted-ca-bundle\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.639521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.639522 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9f6z\" (UniqueName: \"kubernetes.io/projected/608e1cfd-3e3f-4bad-b275-74805261379b-kube-api-access-g9f6z\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.639757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.639553 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/608e1cfd-3e3f-4bad-b275-74805261379b-archives-path\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.639757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.639596 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-service-ca-bundle\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.640006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.639985 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/608e1cfd-3e3f-4bad-b275-74805261379b-archives-path\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.640315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.640294 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-trusted-ca-bundle\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.640401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.640342 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-service-ca-bundle\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.647664 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.647635 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9f6z\" (UniqueName: \"kubernetes.io/projected/608e1cfd-3e3f-4bad-b275-74805261379b-kube-api-access-g9f6z\") pod \"periodic-gathering-cfwcb-f2q7x\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:44.779692 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:44.779651 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:15:45.051626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:45.051530 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cfwcb-f2q7x"] Nov 04 01:15:45.054595 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:15:45.054555 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608e1cfd_3e3f_4bad_b275_74805261379b.slice/crio-c0fad53a676dea94b2b81f7d151a1b1ab9a9ec2332ebfc40f6752862b8f971b7 WatchSource:0}: Error finding container c0fad53a676dea94b2b81f7d151a1b1ab9a9ec2332ebfc40f6752862b8f971b7: Status 404 returned error can't find the container with id c0fad53a676dea94b2b81f7d151a1b1ab9a9ec2332ebfc40f6752862b8f971b7 Nov 04 01:15:45.250165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:45.249970 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" event={"ID":"608e1cfd-3e3f-4bad-b275-74805261379b","Type":"ContainerStarted","Data":"ad3bad95e59c89ff67371c09295834136c9842235a1b5742c85ed1e0dde701ce"} Nov 04 01:15:45.250165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:45.250033 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" event={"ID":"608e1cfd-3e3f-4bad-b275-74805261379b","Type":"ContainerStarted","Data":"c0fad53a676dea94b2b81f7d151a1b1ab9a9ec2332ebfc40f6752862b8f971b7"} Nov 04 01:15:45.266216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:15:45.266159 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" podStartSLOduration=1.26614541 podStartE2EDuration="1.26614541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:15:45.265190009 +0000 UTC m=+1084.236938877" watchObservedRunningTime="2026-11-04 01:15:45.26614541 +0000 UTC m=+1084.237894280" Nov 04 01:16:04.104476 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:16:04.104436 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:16:10.597506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.597474 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2"] Nov 04 01:16:10.601463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.601437 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.612917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.612887 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2"] Nov 04 01:16:10.658389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658359 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.658529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658412 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.658529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658511 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.658606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658564 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8804da-7930-4bee-9769-6910cf9db442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.658649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658627 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8804da-7930-4bee-9769-6910cf9db442-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.658697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658668 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsc7w\" (UniqueName: \"kubernetes.io/projected/8d8804da-7930-4bee-9769-6910cf9db442-kube-api-access-wsc7w\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.658697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658692 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8804da-7930-4bee-9769-6910cf9db442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.658802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658784 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.658851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.658811 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760040 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760077 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760110 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8804da-7930-4bee-9769-6910cf9db442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760158 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8804da-7930-4bee-9769-6910cf9db442-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760181 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsc7w\" (UniqueName: \"kubernetes.io/projected/8d8804da-7930-4bee-9769-6910cf9db442-kube-api-access-wsc7w\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760204 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8804da-7930-4bee-9769-6910cf9db442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760244 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760269 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760355 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760553 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760635 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760704 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760711 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.760879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.760857 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d8804da-7930-4bee-9769-6910cf9db442-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.762696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.762671 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d8804da-7930-4bee-9769-6910cf9db442-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.762804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.762788 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d8804da-7930-4bee-9769-6910cf9db442-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.768756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.768736 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d8804da-7930-4bee-9769-6910cf9db442-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.768907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.768889 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsc7w\" (UniqueName: \"kubernetes.io/projected/8d8804da-7930-4bee-9769-6910cf9db442-kube-api-access-wsc7w\") pod \"maas-default-gateway-openshift-default-75d7c785f-bk4g2\" (UID: \"8d8804da-7930-4bee-9769-6910cf9db442\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:10.917456 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:10.917386 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:11.197372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:11.197345 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2"] Nov 04 01:16:11.198683 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:16:11.198657 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8804da_7930_4bee_9769_6910cf9db442.slice/crio-89a62cd669bde3e5fe5483809645b311b42f5286ce4b7fe42de25cca22bc506d WatchSource:0}: Error finding container 89a62cd669bde3e5fe5483809645b311b42f5286ce4b7fe42de25cca22bc506d: Status 404 returned error can't find the container with id 89a62cd669bde3e5fe5483809645b311b42f5286ce4b7fe42de25cca22bc506d Nov 04 01:16:11.200609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:11.200576 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:11.200713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:11.200645 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:11.200713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:11.200674 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:11.366702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:11.366665 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" event={"ID":"8d8804da-7930-4bee-9769-6910cf9db442","Type":"ContainerStarted","Data":"c6964f103550141520e37801c115cdb962efa494b8f78479be32d82769de239d"} Nov 04 01:16:11.366862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:11.366726 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" event={"ID":"8d8804da-7930-4bee-9769-6910cf9db442","Type":"ContainerStarted","Data":"89a62cd669bde3e5fe5483809645b311b42f5286ce4b7fe42de25cca22bc506d"} Nov 04 01:16:11.389503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:11.389450 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podStartSLOduration=1.3894325570000001 podStartE2EDuration="1.389432557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:11.387214735 +0000 UTC m=+1110.358963606" watchObservedRunningTime="2026-11-04 01:16:11.389432557 +0000 UTC m=+1110.361181432" Nov 04 01:16:11.918061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:11.918032 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:12.919581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:12.919543 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:12.919956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:12.919605 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:16:13.918145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:13.918102 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:13.918317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:13.918172 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:16:14.924160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:14.923591 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:14.924160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:14.923656 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:16:15.918347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:15.918309 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:15.918504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:15.918379 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:16:16.917841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:16.917807 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 04 01:16:16.918262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:16.917858 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 04 01:16:17.921962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:17.921926 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:17.922436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:17.922216 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:17.922890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:17.922871 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" Nov 04 01:16:25.615673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.615577 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq"] Nov 04 01:16:25.621816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.621793 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.635950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.635924 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq"] Nov 04 01:16:25.723002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.722971 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vm9d\" (UniqueName: \"kubernetes.io/projected/5f694985-9c0e-45ab-93b4-fa5ebe379290-kube-api-access-9vm9d\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.723181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.723042 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.723181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.723088 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.723256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.723188 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.723256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.723212 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f694985-9c0e-45ab-93b4-fa5ebe379290-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.723256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.723233 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.723256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.723249 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.723385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.723365 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.723420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.723395 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.823987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.823958 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.823987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.823989 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824008 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824093 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f694985-9c0e-45ab-93b4-fa5ebe379290-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824134 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824162 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824200 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824229 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824306 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vm9d\" (UniqueName: \"kubernetes.io/projected/5f694985-9c0e-45ab-93b4-fa5ebe379290-kube-api-access-9vm9d\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824496 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824647 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824537 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824753 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824832 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f694985-9c0e-45ab-93b4-fa5ebe379290-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.824888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.824850 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.826398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.826378 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.826774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.826757 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.832268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.832247 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vm9d\" (UniqueName: \"kubernetes.io/projected/5f694985-9c0e-45ab-93b4-fa5ebe379290-kube-api-access-9vm9d\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.832356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.832289 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f694985-9c0e-45ab-93b4-fa5ebe379290-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2mvtq\" (UID: \"5f694985-9c0e-45ab-93b4-fa5ebe379290\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:25.939994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:25.939940 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:26.216180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:26.216154 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq"] Nov 04 01:16:26.218146 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:16:26.218092 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f694985_9c0e_45ab_93b4_fa5ebe379290.slice/crio-a000ed6426baba3d91b9e4d2222d3f085471c95c57e4377b4d7735bd292fa5e7 WatchSource:0}: Error finding container a000ed6426baba3d91b9e4d2222d3f085471c95c57e4377b4d7735bd292fa5e7: Status 404 returned error can't find the container with id a000ed6426baba3d91b9e4d2222d3f085471c95c57e4377b4d7735bd292fa5e7 Nov 04 01:16:26.220259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:26.220217 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:26.220374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:26.220296 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:26.220374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:26.220341 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:26.457082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:26.457038 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" event={"ID":"5f694985-9c0e-45ab-93b4-fa5ebe379290","Type":"ContainerStarted","Data":"84696cc8eab3d9b7c1b20fee46a96e8ecce1d2105f296f969d420ba32a7defca"} Nov 04 01:16:26.457280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:26.457096 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" event={"ID":"5f694985-9c0e-45ab-93b4-fa5ebe379290","Type":"ContainerStarted","Data":"a000ed6426baba3d91b9e4d2222d3f085471c95c57e4377b4d7735bd292fa5e7"} Nov 04 01:16:26.477900 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:26.477804 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podStartSLOduration=1.477786474 podStartE2EDuration="1.477786474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:26.477083223 +0000 UTC m=+1125.448832095" watchObservedRunningTime="2026-11-04 01:16:26.477786474 +0000 UTC m=+1125.449535336" Nov 04 01:16:26.940344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:26.940312 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:27.942418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:27.942385 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:16:27.942801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:27.942439 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:16:28.941165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:28.941114 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:16:28.941413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:28.941191 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:16:29.940737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:29.940695 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:16:29.941250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:29.940754 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:16:30.941062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:30.941026 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:16:30.941489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:30.941080 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:16:31.941106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.941070 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 04 01:16:31.941507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.941137 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 04 01:16:31.981206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.981075 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj"] Nov 04 01:16:31.985175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.985151 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj"] Nov 04 01:16:31.985322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.985255 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:31.988854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.988809 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 01:16:31.988980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.988889 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 01:16:31.988980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.988901 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 04 01:16:31.989913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:31.989894 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-njzcd\"" Nov 04 01:16:32.104425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.104387 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51487e32-bffb-43e9-a770-720b3d6d383c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.104598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.104524 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.104598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.104584 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtbv\" (UniqueName: \"kubernetes.io/projected/51487e32-bffb-43e9-a770-720b3d6d383c-kube-api-access-zvtbv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.104711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.104696 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.104746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.104727 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.104841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.104828 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.205946 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.206133 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtbv\" (UniqueName: \"kubernetes.io/projected/51487e32-bffb-43e9-a770-720b3d6d383c-kube-api-access-zvtbv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.206238 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.206271 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.206320 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.206391 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51487e32-bffb-43e9-a770-720b3d6d383c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.206733 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.206769 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.206899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.206881 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.208657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.208628 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51487e32-bffb-43e9-a770-720b3d6d383c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.208955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.208935 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51487e32-bffb-43e9-a770-720b3d6d383c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.214479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.214454 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtbv\" (UniqueName: \"kubernetes.io/projected/51487e32-bffb-43e9-a770-720b3d6d383c-kube-api-access-zvtbv\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-g4dbj\" (UID: \"51487e32-bffb-43e9-a770-720b3d6d383c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.300516 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.300484 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:32.580131 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:32.579194 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj"] Nov 04 01:16:32.583889 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:16:32.582954 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51487e32_bffb_43e9_a770_720b3d6d383c.slice/crio-4d3588855fe8f4b0a6af2920440be0fae45ec0d500d3bf964ba98a9fffc69da6 WatchSource:0}: Error finding container 4d3588855fe8f4b0a6af2920440be0fae45ec0d500d3bf964ba98a9fffc69da6: Status 404 returned error can't find the container with id 4d3588855fe8f4b0a6af2920440be0fae45ec0d500d3bf964ba98a9fffc69da6 Nov 04 01:16:33.197208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:33.197176 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:33.198440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:33.198406 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:33.199302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:33.199237 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" Nov 04 01:16:33.511484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:33.511444 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" event={"ID":"51487e32-bffb-43e9-a770-720b3d6d383c","Type":"ContainerStarted","Data":"4d3588855fe8f4b0a6af2920440be0fae45ec0d500d3bf964ba98a9fffc69da6"} Nov 04 01:16:34.109319 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:16:34.109278 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:16:34.517676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:34.517598 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" event={"ID":"51487e32-bffb-43e9-a770-720b3d6d383c","Type":"ContainerStarted","Data":"9525fa8add73e253a35d951bad36afe00da65269d39d574fbd3e5837c6903a0a"} Nov 04 01:16:34.537311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:34.537248 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podStartSLOduration=1.885297498 podStartE2EDuration="3.53723018s" totalImagesPullingTime="1.651932682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:34.535521223 +0000 UTC m=+1133.507270094" watchObservedRunningTime="2026-11-04 01:16:34.53723018 +0000 UTC m=+1133.508979052" Nov 04 01:16:35.009145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:34.996188 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kt9vs"] Nov 04 01:16:35.035970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.035264 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.035970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.035482 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kt9vs"] Nov 04 01:16:35.044665 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.044638 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 04 01:16:35.198541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.198505 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80b3f88-24db-4e50-aecc-c1c19664e0ae-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.198854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.198829 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.198968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.198904 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-home\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.198968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.198946 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.199086 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.198977 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdrr\" (UniqueName: \"kubernetes.io/projected/f80b3f88-24db-4e50-aecc-c1c19664e0ae-kube-api-access-gxdrr\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.199086 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.199008 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.299956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.299856 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdrr\" (UniqueName: \"kubernetes.io/projected/f80b3f88-24db-4e50-aecc-c1c19664e0ae-kube-api-access-gxdrr\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.299956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.299908 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.300229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.299964 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80b3f88-24db-4e50-aecc-c1c19664e0ae-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.300229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.300039 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.300229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.300097 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-home\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.300229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.300168 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.300538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.300514 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.300634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.300547 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.300634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.300585 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-home\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.300634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.300614 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:35.302872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.302847 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80b3f88-24db-4e50-aecc-c1c19664e0ae-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.303085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.303067 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80b3f88-24db-4e50-aecc-c1c19664e0ae-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.309261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.309242 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdrr\" (UniqueName: \"kubernetes.io/projected/f80b3f88-24db-4e50-aecc-c1c19664e0ae-kube-api-access-gxdrr\") pod \"echo-simulator-kserve-64b6f8f96c-kt9vs\" (UID: \"f80b3f88-24db-4e50-aecc-c1c19664e0ae\") " pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.397882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.397858 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:35.672184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:35.672091 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-kt9vs"] Nov 04 01:16:35.675417 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:16:35.675375 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b3f88_24db_4e50_aecc_c1c19664e0ae.slice/crio-cce5c957cc50077b84d4c1d3487b0ec1d2fe6fbbe0b59b6c3dafe719017918ac WatchSource:0}: Error finding container cce5c957cc50077b84d4c1d3487b0ec1d2fe6fbbe0b59b6c3dafe719017918ac: Status 404 returned error can't find the container with id cce5c957cc50077b84d4c1d3487b0ec1d2fe6fbbe0b59b6c3dafe719017918ac Nov 04 01:16:36.548989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.548519 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" event={"ID":"f80b3f88-24db-4e50-aecc-c1c19664e0ae","Type":"ContainerStarted","Data":"cce5c957cc50077b84d4c1d3487b0ec1d2fe6fbbe0b59b6c3dafe719017918ac"} Nov 04 01:16:36.606143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.605356 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p"] Nov 04 01:16:36.656145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.649728 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p"] Nov 04 01:16:36.656145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.650578 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.707869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.707814 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 04 01:16:36.745811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.745784 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.745968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.745835 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.745968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.745861 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.746090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.745963 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg248\" (UniqueName: \"kubernetes.io/projected/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-kube-api-access-lg248\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.746090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.746013 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.746200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.746084 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.847314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.847202 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.847463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.847320 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.847463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.847384 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.847582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.847484 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.847582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.847540 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg248\" (UniqueName: \"kubernetes.io/projected/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-kube-api-access-lg248\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.847684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.847584 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.848023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.847994 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.848149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.848033 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.848332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.848310 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.871860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.871801 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg248\" (UniqueName: \"kubernetes.io/projected/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-kube-api-access-lg248\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.877146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.873716 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:36.897499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:36.897418 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f2221e-2cb7-4513-a72c-e6e4156d6b73-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p\" (UID: \"e1f2221e-2cb7-4513-a72c-e6e4156d6b73\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:37.056155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:37.051293 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:37.399169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:37.399068 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p"] Nov 04 01:16:37.404134 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:16:37.404088 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f2221e_2cb7_4513_a72c_e6e4156d6b73.slice/crio-42eb6014ea9c4b9b3c573c3328303973d4317a3b0006ad0d57089e3d6803a40b WatchSource:0}: Error finding container 42eb6014ea9c4b9b3c573c3328303973d4317a3b0006ad0d57089e3d6803a40b: Status 404 returned error can't find the container with id 42eb6014ea9c4b9b3c573c3328303973d4317a3b0006ad0d57089e3d6803a40b Nov 04 01:16:37.555921 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:37.555889 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" event={"ID":"e1f2221e-2cb7-4513-a72c-e6e4156d6b73","Type":"ContainerStarted","Data":"42eb6014ea9c4b9b3c573c3328303973d4317a3b0006ad0d57089e3d6803a40b"} Nov 04 01:16:38.567142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:38.566182 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" event={"ID":"e1f2221e-2cb7-4513-a72c-e6e4156d6b73","Type":"ContainerStarted","Data":"ef9e930834068a93d1c5da5fe979c63c2fb1cd95542e886d0c2dcf29727504f1"} Nov 04 01:16:38.614145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:38.601107 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podStartSLOduration=2.299794489 podStartE2EDuration="2.601089151s" totalImagesPullingTime="301.294662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:38.597494376 +0000 UTC m=+1137.569243248" watchObservedRunningTime="2026-11-04 01:16:38.601089151 +0000 UTC m=+1137.572838023" Nov 04 01:16:39.053224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.053188 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:39.587780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.587673 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" event={"ID":"f80b3f88-24db-4e50-aecc-c1c19664e0ae","Type":"ContainerStarted","Data":"9f3c999a84db5b8710fbb7a5a998584d791a2895570ab4bb2623ed75f61c76b5"} Nov 04 01:16:39.610212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.608875 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" podStartSLOduration=2.349956075 podStartE2EDuration="5.608860275s" totalImagesPullingTime="3.2589042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:39.607987618 +0000 UTC m=+1138.579736490" watchObservedRunningTime="2026-11-04 01:16:39.608860275 +0000 UTC m=+1138.580609147" Nov 04 01:16:39.812343 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.812299 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s"] Nov 04 01:16:39.816184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.816160 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.821185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.821160 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s"] Nov 04 01:16:39.823989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.823965 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 04 01:16:39.858059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.857987 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.858059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.858029 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.858275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.858153 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.858275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.858181 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a1d019-a504-4116-929d-33ad787904e5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.858275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.858205 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.858430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.858382 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgxr\" (UniqueName: \"kubernetes.io/projected/56a1d019-a504-4116-929d-33ad787904e5-kube-api-access-khgxr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.959718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.959681 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.959908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.959772 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.959908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.959812 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a1d019-a504-4116-929d-33ad787904e5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.959908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.959848 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.960066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.959913 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgxr\" (UniqueName: \"kubernetes.io/projected/56a1d019-a504-4116-929d-33ad787904e5-kube-api-access-khgxr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.960066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.959958 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.960366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.960334 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.960522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.960502 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.960646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.960623 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.962219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.962192 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a1d019-a504-4116-929d-33ad787904e5-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.962925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.962903 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a1d019-a504-4116-929d-33ad787904e5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:39.968414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:39.968388 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgxr\" (UniqueName: \"kubernetes.io/projected/56a1d019-a504-4116-929d-33ad787904e5-kube-api-access-khgxr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s\" (UID: \"56a1d019-a504-4116-929d-33ad787904e5\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:40.134881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.134788 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:40.399362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.399266 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:40.422925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.422889 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s"] Nov 04 01:16:40.427424 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:16:40.427399 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a1d019_a504_4116_929d_33ad787904e5.slice/crio-bc01d7a817e697e19c5c27ad6090efc76d72d89acbeec3ac88308bfefe17b3c7 WatchSource:0}: Error finding container bc01d7a817e697e19c5c27ad6090efc76d72d89acbeec3ac88308bfefe17b3c7: Status 404 returned error can't find the container with id bc01d7a817e697e19c5c27ad6090efc76d72d89acbeec3ac88308bfefe17b3c7 Nov 04 01:16:40.595729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.595692 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" event={"ID":"56a1d019-a504-4116-929d-33ad787904e5","Type":"ContainerStarted","Data":"bc01d7a817e697e19c5c27ad6090efc76d72d89acbeec3ac88308bfefe17b3c7"} Nov 04 01:16:40.637703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.637670 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt"] Nov 04 01:16:40.643263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.643234 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.656398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.656374 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt"] Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.670978 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a741a-6c41-4981-a205-f01665ad111e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.671043 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a741a-6c41-4981-a205-f01665ad111e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.671097 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a741a-6c41-4981-a205-f01665ad111e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.671153 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.671181 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.671292 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dkgm\" (UniqueName: \"kubernetes.io/projected/ed9a741a-6c41-4981-a205-f01665ad111e-kube-api-access-2dkgm\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.671345 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.671391 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.672263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.672047 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773490 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773572 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773619 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a741a-6c41-4981-a205-f01665ad111e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773663 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a741a-6c41-4981-a205-f01665ad111e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a741a-6c41-4981-a205-f01665ad111e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773747 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773776 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773886 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dkgm\" (UniqueName: \"kubernetes.io/projected/ed9a741a-6c41-4981-a205-f01665ad111e-kube-api-access-2dkgm\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.773926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.773921 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.775352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.774836 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.775352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.775304 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.775529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.775385 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.775746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.775706 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a741a-6c41-4981-a205-f01665ad111e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.776716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.776691 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.777342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.777316 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a741a-6c41-4981-a205-f01665ad111e-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.778518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.778494 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a741a-6c41-4981-a205-f01665ad111e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.801310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.801281 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dkgm\" (UniqueName: \"kubernetes.io/projected/ed9a741a-6c41-4981-a205-f01665ad111e-kube-api-access-2dkgm\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.801423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.801354 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a741a-6c41-4981-a205-f01665ad111e-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-xkmtt\" (UID: \"ed9a741a-6c41-4981-a205-f01665ad111e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:40.976905 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:40.976759 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:41.445828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:41.445803 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt"] Nov 04 01:16:41.535465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:41.535420 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:41.539280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:41.539245 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:41.539453 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:41.539435 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:16:41.678719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:41.678680 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" event={"ID":"56a1d019-a504-4116-929d-33ad787904e5","Type":"ContainerStarted","Data":"9225c93d2c8888b13bc03d659eee5d8dc89aae3b859966d78878cab93ae18571"} Nov 04 01:16:41.682964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:41.682938 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" event={"ID":"ed9a741a-6c41-4981-a205-f01665ad111e","Type":"ContainerStarted","Data":"ac04252fbb148060a591e5eeb4d9e0173dde0f5ca8fdb1c85d9600233baef286"} Nov 04 01:16:41.728964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:41.728666 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podStartSLOduration=2.421423963 podStartE2EDuration="2.728648163s" totalImagesPullingTime="307.2242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:41.728362481 +0000 UTC m=+1140.700111353" watchObservedRunningTime="2026-11-04 01:16:41.728648163 +0000 UTC m=+1140.700397035" Nov 04 01:16:42.138164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.135102 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:42.302041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.301943 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:42.330041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.330008 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:42.700216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.691237 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" event={"ID":"ed9a741a-6c41-4981-a205-f01665ad111e","Type":"ContainerStarted","Data":"4b4a222552a8d9d170623b2b0691d4111a98372efff20a7d0b8a1c53f2bdbe05"} Nov 04 01:16:42.716767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.716738 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:16:42.728012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.727962 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podStartSLOduration=2.727948987 podStartE2EDuration="2.727948987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:42.725246938 +0000 UTC m=+1141.696995813" watchObservedRunningTime="2026-11-04 01:16:42.727948987 +0000 UTC m=+1141.699697858" Nov 04 01:16:42.979397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.978198 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:42.979725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.979688 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:16:42.979851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:42.979755 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 04 01:16:43.978144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:43.977489 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:16:43.978144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:43.977551 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 04 01:16:44.991169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:44.980502 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:16:44.991169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:44.980570 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 04 01:16:45.404199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:45.403260 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:45.452144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:45.450538 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:45.796430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:45.796398 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" Nov 04 01:16:46.080660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:46.080567 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:16:46.081248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:46.081214 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 04 01:16:46.980589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:46.980545 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:16:46.980893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:46.980860 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 04 01:16:47.057485 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:47.057451 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:47.090611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:47.090582 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:47.826432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:47.826395 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:16:47.978572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:47.978536 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:16:47.978874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:47.978827 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 04 01:16:48.978143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:48.977717 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:16:48.978676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:48.978624 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 04 01:16:49.248144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:49.247302 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:49.248144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:49.247387 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:49.977632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:49.977598 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 04 01:16:49.977935 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:49.977889 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 04 01:16:50.135387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:50.135351 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:50.161143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:50.156670 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:50.838512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:50.838483 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:16:52.018278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:52.018239 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:52.018683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:52.018295 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:53.018643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:53.018609 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:53.019182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:53.018664 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:54.060316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.060262 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:16:54.060838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.060345 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:16:54.746637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.746595 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7457bc8fc9-m4sb2"] Nov 04 01:16:54.751755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.751725 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" Nov 04 01:16:54.768746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.768720 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 04 01:16:54.769006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.768987 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xvwz\"" Nov 04 01:16:54.780511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.771941 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7457bc8fc9-m4sb2"] Nov 04 01:16:54.845320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.845296 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9622120-9f16-41e2-8a70-77f05616c703-tls-cert\") pod \"authorino-7457bc8fc9-m4sb2\" (UID: \"d9622120-9f16-41e2-8a70-77f05616c703\") " pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" Nov 04 01:16:54.845474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.845337 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zjmd\" (UniqueName: \"kubernetes.io/projected/d9622120-9f16-41e2-8a70-77f05616c703-kube-api-access-6zjmd\") pod \"authorino-7457bc8fc9-m4sb2\" (UID: \"d9622120-9f16-41e2-8a70-77f05616c703\") " pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" Nov 04 01:16:54.946302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.946262 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zjmd\" (UniqueName: \"kubernetes.io/projected/d9622120-9f16-41e2-8a70-77f05616c703-kube-api-access-6zjmd\") pod \"authorino-7457bc8fc9-m4sb2\" (UID: \"d9622120-9f16-41e2-8a70-77f05616c703\") " pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" Nov 04 01:16:54.946442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.946387 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9622120-9f16-41e2-8a70-77f05616c703-tls-cert\") pod \"authorino-7457bc8fc9-m4sb2\" (UID: \"d9622120-9f16-41e2-8a70-77f05616c703\") " pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" Nov 04 01:16:54.948909 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.948880 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9622120-9f16-41e2-8a70-77f05616c703-tls-cert\") pod \"authorino-7457bc8fc9-m4sb2\" (UID: \"d9622120-9f16-41e2-8a70-77f05616c703\") " pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" Nov 04 01:16:54.956365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.956339 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zjmd\" (UniqueName: \"kubernetes.io/projected/d9622120-9f16-41e2-8a70-77f05616c703-kube-api-access-6zjmd\") pod \"authorino-7457bc8fc9-m4sb2\" (UID: \"d9622120-9f16-41e2-8a70-77f05616c703\") " pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" Nov 04 01:16:54.987981 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.987949 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:54.988437 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.988412 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:54.989416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:54.989387 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" Nov 04 01:16:55.069982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:55.069902 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" Nov 04 01:16:55.344113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:55.344094 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7457bc8fc9-m4sb2"] Nov 04 01:16:55.346408 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:16:55.346377 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9622120_9f16_41e2_8a70_77f05616c703.slice/crio-d80d6493c6379ed1b61338eb2b990947dd39aabd26e1571ed20943b3558a63d3 WatchSource:0}: Error finding container d80d6493c6379ed1b61338eb2b990947dd39aabd26e1571ed20943b3558a63d3: Status 404 returned error can't find the container with id d80d6493c6379ed1b61338eb2b990947dd39aabd26e1571ed20943b3558a63d3 Nov 04 01:16:55.855514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:55.855475 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" event={"ID":"d9622120-9f16-41e2-8a70-77f05616c703","Type":"ContainerStarted","Data":"d80d6493c6379ed1b61338eb2b990947dd39aabd26e1571ed20943b3558a63d3"} Nov 04 01:16:57.876342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:57.876301 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" event={"ID":"d9622120-9f16-41e2-8a70-77f05616c703","Type":"ContainerStarted","Data":"e8975b1ac2eb5154f7b44decc0cd6099c0173ef6f6f1949f86300107eb345399"} Nov 04 01:16:57.895457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:16:57.895408 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7457bc8fc9-m4sb2" podStartSLOduration=1.983417616 podStartE2EDuration="3.895395808s" totalImagesPullingTime="1.911978192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:16:57.893566095 +0000 UTC m=+1156.865314967" watchObservedRunningTime="2026-11-04 01:16:57.895395808 +0000 UTC m=+1156.867144678" Nov 04 01:17:04.105644 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:17:04.105600 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:17:19.002705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:19.002670 5051 generic.go:356] "Generic (PLEG): container finished" podID="608e1cfd-3e3f-4bad-b275-74805261379b" containerID="ad3bad95e59c89ff67371c09295834136c9842235a1b5742c85ed1e0dde701ce" exitCode=0 Nov 04 01:17:19.003088 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:19.002742 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" event={"ID":"608e1cfd-3e3f-4bad-b275-74805261379b","Type":"ContainerDied","Data":"ad3bad95e59c89ff67371c09295834136c9842235a1b5742c85ed1e0dde701ce"} Nov 04 01:17:20.154752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.154726 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:17:20.288817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.288736 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-service-ca-bundle\") pod \"608e1cfd-3e3f-4bad-b275-74805261379b\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " Nov 04 01:17:20.288980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.288835 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608e1cfd-3e3f-4bad-b275-74805261379b-kube-api-access-g9f6z\" (UniqueName: \"kubernetes.io/projected/608e1cfd-3e3f-4bad-b275-74805261379b-kube-api-access-g9f6z\") pod \"608e1cfd-3e3f-4bad-b275-74805261379b\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " Nov 04 01:17:20.288980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.288888 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608e1cfd-3e3f-4bad-b275-74805261379b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/608e1cfd-3e3f-4bad-b275-74805261379b-archives-path\") pod \"608e1cfd-3e3f-4bad-b275-74805261379b\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " Nov 04 01:17:20.289094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.289042 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-trusted-ca-bundle\") pod \"608e1cfd-3e3f-4bad-b275-74805261379b\" (UID: \"608e1cfd-3e3f-4bad-b275-74805261379b\") " Nov 04 01:17:20.289163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.289086 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-service-ca-bundle" pod "608e1cfd-3e3f-4bad-b275-74805261379b" (UID: "608e1cfd-3e3f-4bad-b275-74805261379b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:17:20.289359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.289327 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608e1cfd-3e3f-4bad-b275-74805261379b-archives-path" pod "608e1cfd-3e3f-4bad-b275-74805261379b" (UID: "608e1cfd-3e3f-4bad-b275-74805261379b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:17:20.289480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.289416 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-trusted-ca-bundle" pod "608e1cfd-3e3f-4bad-b275-74805261379b" (UID: "608e1cfd-3e3f-4bad-b275-74805261379b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:17:20.289480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.289431 5051 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-service-ca-bundle\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:20.289480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.289455 5051 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/608e1cfd-3e3f-4bad-b275-74805261379b-archives-path\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:20.291065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.291047 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608e1cfd-3e3f-4bad-b275-74805261379b-kube-api-access-g9f6z" pod "608e1cfd-3e3f-4bad-b275-74805261379b" (UID: "608e1cfd-3e3f-4bad-b275-74805261379b"). InnerVolumeSpecName "kube-api-access-g9f6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:17:20.390299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.390266 5051 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/608e1cfd-3e3f-4bad-b275-74805261379b-trusted-ca-bundle\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:20.390299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:20.390293 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9f6z\" (UniqueName: \"kubernetes.io/projected/608e1cfd-3e3f-4bad-b275-74805261379b-kube-api-access-g9f6z\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:17:21.015210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:21.015183 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" Nov 04 01:17:21.015210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:21.015186 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cfwcb-f2q7x" event={"ID":"608e1cfd-3e3f-4bad-b275-74805261379b","Type":"ContainerDied","Data":"c0fad53a676dea94b2b81f7d151a1b1ab9a9ec2332ebfc40f6752862b8f971b7"} Nov 04 01:17:21.015414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:21.015265 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0fad53a676dea94b2b81f7d151a1b1ab9a9ec2332ebfc40f6752862b8f971b7" Nov 04 01:17:34.104049 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:17:34.104005 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:17:41.316205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:41.316171 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:17:41.319715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:41.319689 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:17:56.745359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.745279 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z"] Nov 04 01:17:56.745785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.745767 5051 cpu_manager.go:414] "containerMap: removing container" podUID="608e1cfd-3e3f-4bad-b275-74805261379b" containerName="insights-gathering" Nov 04 01:17:56.745854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.745785 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608e1cfd-3e3f-4bad-b275-74805261379b" containerName="insights-gathering" Nov 04 01:17:56.745909 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.745888 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="608e1cfd-3e3f-4bad-b275-74805261379b" containerName="insights-gathering" Nov 04 01:17:56.749381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.749357 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.768580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.768558 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9c8bf5-openshift-default-dockercfg-4ks7x\"" Nov 04 01:17:56.778451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.778424 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z"] Nov 04 01:17:56.846305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846275 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.846454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846321 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.846454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846350 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.846454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846421 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.846454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846447 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mhns\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-kube-api-access-7mhns\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.846587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846526 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.846587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846548 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.846587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846564 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd3792-4e02-4f79-9049-8d4bd4973d48-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.846714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.846692 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.917410 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.917374 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f"] Nov 04 01:17:56.921762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.921740 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:56.931302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.931282 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-099dd9-openshift-default-dockercfg-ntzzj\"" Nov 04 01:17:56.942706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.942665 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f"] Nov 04 01:17:56.947593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947567 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.947727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947630 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.947727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947664 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.947727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947681 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.947727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947704 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.947965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947727 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mhns\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-kube-api-access-7mhns\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.947965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947766 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.947965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947804 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.947965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.947827 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd3792-4e02-4f79-9049-8d4bd4973d48-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.948507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.948483 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd3792-4e02-4f79-9049-8d4bd4973d48-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.949478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.949452 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.949593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.949471 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.949593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.949531 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.950913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.950883 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.951824 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.951805 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.952083 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.952050 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.964525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.964504 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:56.964963 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:56.964944 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mhns\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-kube-api-access-7mhns\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:57.045861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.045777 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf"] Nov 04 01:17:57.049109 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049068 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.049304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049179 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.055199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049677 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.055199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049731 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hp9\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-kube-api-access-z9hp9\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.055199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049824 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.055199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049860 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.055199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049885 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.055199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049933 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.055199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.049963 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.058727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.058687 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.062008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.061974 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf"] Nov 04 01:17:57.067389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.067291 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-499ddd-openshift-default-dockercfg-fb8nq\"" Nov 04 01:17:57.072304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.068273 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:57.118646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.118605 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx"] Nov 04 01:17:57.123321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.123275 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.142144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.139539 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx"] Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.150679 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.150726 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.150754 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.150798 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.150826 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.150863 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.154336 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.154580 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.150899 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155275 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155322 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155379 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155418 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155457 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155490 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155516 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hp9\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-kube-api-access-z9hp9\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155550 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlv8\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-kube-api-access-wwlv8\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155601 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155655 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.157082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.155691 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.158795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.158643 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ca55c1-openshift-default-dockercfg-t5hrc\"" Nov 04 01:17:57.160141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.159678 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.160141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.159749 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.161030 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.160830 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.162853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.162771 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.165268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.165207 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.170483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.170439 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.179798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.179767 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hp9\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-kube-api-access-z9hp9\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.199157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.198474 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb"] Nov 04 01:17:57.204284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.203722 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.216302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.216278 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb"] Nov 04 01:17:57.237335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.237313 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:57.237556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.237325 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ee876a-openshift-default-dockercfg-v2pll\"" Nov 04 01:17:57.257469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.257441 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4067319-b8e0-4ec9-8356-93c60b083658-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.257699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.257659 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4067319-b8e0-4ec9-8356-93c60b083658-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.257920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.257901 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.258069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.258051 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.258218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.258202 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.258417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.258399 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.258563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.258545 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.258707 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.258690 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.258880 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.258830 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.259055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.258992 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.259283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.259247 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.259464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.259409 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.259716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.259597 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.259716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.259681 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlv8\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-kube-api-access-wwlv8\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.260017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.259982 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.260661 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.260582 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.260661 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.260629 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfptb\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-kube-api-access-qfptb\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.261001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.260919 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.262508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.262489 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.263983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.263271 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.263983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.263512 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.264241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.264143 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.265422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.265376 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.265568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.265514 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.265773 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.265730 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.276803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.276744 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.283579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.283535 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf"] Nov 04 01:17:57.288794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.288765 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlv8\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-kube-api-access-wwlv8\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.294268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.289860 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.300425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.300106 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf"] Nov 04 01:17:57.300425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.300322 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-586541-openshift-default-dockercfg-tgz2p\"" Nov 04 01:17:57.362392 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362360 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.362637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362615 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.362853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362787 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.362853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362797 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.362853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362832 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.363056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362870 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.363056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362897 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.363056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362934 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.363056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.362964 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.363056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363006 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.363056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363031 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.363368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363071 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.363368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363147 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7vf\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-kube-api-access-mw7vf\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.363368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363200 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.363368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363237 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.363368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363291 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.363368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363348 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfptb\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-kube-api-access-qfptb\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.363721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363395 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4067319-b8e0-4ec9-8356-93c60b083658-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.363721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.363443 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4067319-b8e0-4ec9-8356-93c60b083658-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.364783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.364162 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.364783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.364209 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4067319-b8e0-4ec9-8356-93c60b083658-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.364783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.364462 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.364783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.364653 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.365041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.364827 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.368538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.368461 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4067319-b8e0-4ec9-8356-93c60b083658-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.373208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.373191 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2"] Nov 04 01:17:57.374911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.374876 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfptb\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-kube-api-access-qfptb\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.376371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.376349 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.379549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.379532 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.382938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.382917 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:57.400612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.400544 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2"] Nov 04 01:17:57.401591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.401569 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z"] Nov 04 01:17:57.403207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.403168 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d7e45d-openshift-default-dockercfg-rl5m5\"" Nov 04 01:17:57.422267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.422235 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.422348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.422301 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.422348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.422331 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.463396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.463169 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:57.467734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.467701 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.484183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.467196 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.484383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.484360 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.484493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.484480 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.484613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.484600 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.484711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.484695 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.484851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.484830 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.484969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.484955 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.485076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.485059 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.485271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.485236 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5de76c-942c-4e1c-bd03-367aca840dbd-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.485488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.485444 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7vf\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-kube-api-access-mw7vf\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.485667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.485628 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwc9c\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-kube-api-access-qwc9c\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.486589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.486556 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.486788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.486718 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.487464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.487405 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c27e987-5d66-42a9-8ac3-3a377697bc18-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500584 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500648 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500697 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500735 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500760 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500815 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500845 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500888 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500921 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500944 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.500984 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.501015 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.501038 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.501068 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvgh\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-kube-api-access-6kvgh\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.501138 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.501808 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.502287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.502103 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.506756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.506704 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.522910 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.522879 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7vf\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-kube-api-access-mw7vf\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.539365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.530704 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.539365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.532543 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.580152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.580088 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f"] Nov 04 01:17:57.588509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.588463 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.588638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.588561 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.588638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.588605 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.602832 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.602933 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.602991 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603025 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603077 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5de76c-942c-4e1c-bd03-367aca840dbd-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603105 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwc9c\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-kube-api-access-qwc9c\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603172 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c27e987-5d66-42a9-8ac3-3a377697bc18-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603229 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603260 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603305 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603335 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603374 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603406 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603431 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603466 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603495 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603520 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.603546 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvgh\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-kube-api-access-6kvgh\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.604326 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.604680 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.604951 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.605322 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.605490 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.605772 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.605787 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.612232 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5de76c-942c-4e1c-bd03-367aca840dbd-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.613105 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c27e987-5d66-42a9-8ac3-3a377697bc18-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.614986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.613437 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.616362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.616340 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.620155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.619151 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.623171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.623101 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.624372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.624349 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.624562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.624544 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.624781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.624758 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvgh\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-kube-api-access-6kvgh\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.625984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.625927 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwc9c\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-kube-api-access-qwc9c\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.626891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.626869 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:57.707490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.707458 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf"] Nov 04 01:17:57.709238 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:17:57.709212 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9b76a8_cfca_4c94_b8f1_25d0ca7b35d6.slice/crio-536573e70935305a7a758a0c8b86cec6caf1444da92da3cc5707b6358438f424 WatchSource:0}: Error finding container 536573e70935305a7a758a0c8b86cec6caf1444da92da3cc5707b6358438f424: Status 404 returned error can't find the container with id 536573e70935305a7a758a0c8b86cec6caf1444da92da3cc5707b6358438f424 Nov 04 01:17:57.711599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.711566 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.711712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.711652 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.711712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.711683 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.719803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.719776 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:57.832595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.832352 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:57.861674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.861420 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx"] Nov 04 01:17:57.868544 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:17:57.867601 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4067319_b8e0_4ec9_8356_93c60b083658.slice/crio-a4b240984306c0189ad40140445e2361c1046cf7e21b0e8b5b2f180532497437 WatchSource:0}: Error finding container a4b240984306c0189ad40140445e2361c1046cf7e21b0e8b5b2f180532497437: Status 404 returned error can't find the container with id a4b240984306c0189ad40140445e2361c1046cf7e21b0e8b5b2f180532497437 Nov 04 01:17:57.873458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.871838 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.873458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.871940 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.873458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.871981 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:57.925582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:57.925314 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:58.031688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.031650 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2"] Nov 04 01:17:58.045433 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:17:58.045381 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5de76c_942c_4e1c_bd03_367aca840dbd.slice/crio-11c3f1be507f4f30dfd21b4567e2c0798f8c59b080e18a8ad627aabe85ddb8b4 WatchSource:0}: Error finding container 11c3f1be507f4f30dfd21b4567e2c0798f8c59b080e18a8ad627aabe85ddb8b4: Status 404 returned error can't find the container with id 11c3f1be507f4f30dfd21b4567e2c0798f8c59b080e18a8ad627aabe85ddb8b4 Nov 04 01:17:58.050821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.049825 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.050821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.049911 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.050821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.049956 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.185536 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.185360 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb"] Nov 04 01:17:58.193241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.193181 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.193365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.193310 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.193430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.193398 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.226449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.226419 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" event={"ID":"a3170ff2-51dd-4c54-a3ea-8494c86040c9","Type":"ContainerStarted","Data":"f06f47bebecad8023b775b66bde8b04a32dd87c0fca4975245392631b236b3c1"} Nov 04 01:17:58.228433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.228409 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" event={"ID":"35fd3792-4e02-4f79-9049-8d4bd4973d48","Type":"ContainerStarted","Data":"1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c"} Nov 04 01:17:58.228573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.228555 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" event={"ID":"35fd3792-4e02-4f79-9049-8d4bd4973d48","Type":"ContainerStarted","Data":"aaec5f62d603fdc064b6eeab51a8e6baa6046aeccb42e7d051ee1f9804102086"} Nov 04 01:17:58.239335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.237417 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" event={"ID":"5d5de76c-942c-4e1c-bd03-367aca840dbd","Type":"ContainerStarted","Data":"b518baeae647a06d1a41d7f38cf8ca6dad71b7d72250c7ab94a71474e0c455c8"} Nov 04 01:17:58.239335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.237467 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" event={"ID":"5d5de76c-942c-4e1c-bd03-367aca840dbd","Type":"ContainerStarted","Data":"11c3f1be507f4f30dfd21b4567e2c0798f8c59b080e18a8ad627aabe85ddb8b4"} Nov 04 01:17:58.240698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.240442 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" event={"ID":"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6","Type":"ContainerStarted","Data":"96c6c0a31fb77cfe64fe167823a1e8511fe0d916b6e4f81505c345905878c5e6"} Nov 04 01:17:58.240698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.240495 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" event={"ID":"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6","Type":"ContainerStarted","Data":"536573e70935305a7a758a0c8b86cec6caf1444da92da3cc5707b6358438f424"} Nov 04 01:17:58.242692 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.242666 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" event={"ID":"d4067319-b8e0-4ec9-8356-93c60b083658","Type":"ContainerStarted","Data":"b1350ae90d59a02fae6f1c684aa0b9988173658e9280448e9c8136f6ecfec681"} Nov 04 01:17:58.242804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.242715 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" event={"ID":"d4067319-b8e0-4ec9-8356-93c60b083658","Type":"ContainerStarted","Data":"a4b240984306c0189ad40140445e2361c1046cf7e21b0e8b5b2f180532497437"} Nov 04 01:17:58.246837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.246818 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" event={"ID":"859aa7a6-7b2f-4600-b426-7b89f7416ff0","Type":"ContainerStarted","Data":"b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab"} Nov 04 01:17:58.246967 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.246956 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" event={"ID":"859aa7a6-7b2f-4600-b426-7b89f7416ff0","Type":"ContainerStarted","Data":"795125514a92d34e6205af3c4b935507ea8710e780e57de1c259f452ea858f2c"} Nov 04 01:17:58.254715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.254665 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podStartSLOduration=2.254648487 podStartE2EDuration="2.254648487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:58.250098511 +0000 UTC m=+1217.221847384" watchObservedRunningTime="2026-11-04 01:17:58.254648487 +0000 UTC m=+1217.226397358" Nov 04 01:17:58.276513 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.274866 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podStartSLOduration=2.274849571 podStartE2EDuration="2.274849571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:58.270161877 +0000 UTC m=+1217.241910748" watchObservedRunningTime="2026-11-04 01:17:58.274849571 +0000 UTC m=+1217.246598510" Nov 04 01:17:58.276766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.276743 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf"] Nov 04 01:17:58.280377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.280342 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.280560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.280541 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.280673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.280658 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:17:58.298680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.298616 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podStartSLOduration=1.298597863 podStartE2EDuration="1.298597863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:58.295203948 +0000 UTC m=+1217.266952821" watchObservedRunningTime="2026-11-04 01:17:58.298597863 +0000 UTC m=+1217.270346727" Nov 04 01:17:58.325458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.323609 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podStartSLOduration=1.323592495 podStartE2EDuration="1.323592495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:58.320774602 +0000 UTC m=+1217.292523477" watchObservedRunningTime="2026-11-04 01:17:58.323592495 +0000 UTC m=+1217.295341366" Nov 04 01:17:58.383644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.383574 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:58.393727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.393503 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:58.435577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.435224 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podStartSLOduration=1.4352020109999999 podStartE2EDuration="1.435202011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:58.347327286 +0000 UTC m=+1217.319076157" watchObservedRunningTime="2026-11-04 01:17:58.435202011 +0000 UTC m=+1217.406950883" Nov 04 01:17:58.471291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.471214 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:58.720948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:58.720867 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:59.072290 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.072256 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:59.077987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.077961 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:59.237489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.237452 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:59.242757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.242733 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:59.253703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.253672 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" event={"ID":"a3170ff2-51dd-4c54-a3ea-8494c86040c9","Type":"ContainerStarted","Data":"f27228c0ef1e0692373eebc2f18ef46d8f2b15d15eed76bcb07e5a5259018091"} Nov 04 01:17:59.255346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.255286 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" event={"ID":"9c27e987-5d66-42a9-8ac3-3a377697bc18","Type":"ContainerStarted","Data":"9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e"} Nov 04 01:17:59.255346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.255336 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" event={"ID":"9c27e987-5d66-42a9-8ac3-3a377697bc18","Type":"ContainerStarted","Data":"b6cd7c219d0ba51ff3c0cbcae114e5f26a3a358c0441036e99fe3f69b7ee6710"} Nov 04 01:17:59.256149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.256096 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:59.256308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.256167 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:59.256308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.256193 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:59.256654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.256630 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:17:59.256748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.256686 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:17:59.257158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.257112 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:17:59.328470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.328363 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podStartSLOduration=2.328344757 podStartE2EDuration="2.328344757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:59.326554191 +0000 UTC m=+1218.298303059" watchObservedRunningTime="2026-11-04 01:17:59.328344757 +0000 UTC m=+1218.300093629" Nov 04 01:17:59.348586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.348538 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podStartSLOduration=2.348524448 podStartE2EDuration="2.348524448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:17:59.347932857 +0000 UTC m=+1218.319681752" watchObservedRunningTime="2026-11-04 01:17:59.348524448 +0000 UTC m=+1218.320273318" Nov 04 01:17:59.469871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.469844 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:17:59.726152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.726054 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:17:59.835288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.835259 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:59.841258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.841232 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:17:59.926596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.926568 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:17:59.931966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:17:59.931937 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:18:00.261436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:00.261403 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:18:00.262397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:00.262370 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:18:00.262542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:00.262506 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:18:00.262613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:00.262560 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:18:00.262613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:00.262596 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:18:00.262613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:00.262613 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:18:00.263140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:00.263101 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:18:00.263477 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:00.263465 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:18:04.107282 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:18:04.107241 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:18:15.328160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:15.321968 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:15.328160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:15.322055 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:27.817249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:27.817205 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c"] Nov 04 01:18:27.951384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:27.947713 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:27.982959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:27.982907 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c"] Nov 04 01:18:28.010093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.008753 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"openshift-service-ca.crt\"" Nov 04 01:18:28.010093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.008827 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"default-dockercfg-vczgh\"" Nov 04 01:18:28.010093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.009031 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"kube-root-ca.crt\"" Nov 04 01:18:28.010093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.009097 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-499ddd\"/\"simulator-w6-499ddd-kserve-self-signed-certs\"" Nov 04 01:18:28.036395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.035805 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-dshm\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.036395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.035873 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-model-cache\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.036395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.035985 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28707278-956e-4312-a887-be2241f79f79-tls-certs\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.036395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.036029 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpsp\" (UniqueName: \"kubernetes.io/projected/28707278-956e-4312-a887-be2241f79f79-kube-api-access-2jpsp\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.036395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.036259 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-home\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.036395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.036290 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-tmp-dir\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.140186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.138473 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-home\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.140186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.138611 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-tmp-dir\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.140186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.138655 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-dshm\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.140186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.138700 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-model-cache\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.140186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.138825 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28707278-956e-4312-a887-be2241f79f79-tls-certs\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.140186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.138873 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpsp\" (UniqueName: \"kubernetes.io/projected/28707278-956e-4312-a887-be2241f79f79-kube-api-access-2jpsp\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.140186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.139747 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-home\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.141890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.140079 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-tmp-dir\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.141890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.141053 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-model-cache\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.150859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.150804 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28707278-956e-4312-a887-be2241f79f79-tls-certs\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.164470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.164396 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-dshm\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.169896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.169829 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpsp\" (UniqueName: \"kubernetes.io/projected/28707278-956e-4312-a887-be2241f79f79-kube-api-access-2jpsp\") pod \"simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.362194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.361851 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:28.787487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:28.787441 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c"] Nov 04 01:18:29.333390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:29.312864 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" event={"ID":"28707278-956e-4312-a887-be2241f79f79","Type":"ContainerStarted","Data":"39cdc41ef49e5222314d506bf964e9266d7748547d44e2a5cc4c044f9c4a5623"} Nov 04 01:18:29.333390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:29.312926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" event={"ID":"28707278-956e-4312-a887-be2241f79f79","Type":"ContainerStarted","Data":"5679ab25ba0d8d449471fc8e241c6952b2c727693545d3099894a719fd297957"} Nov 04 01:18:29.336744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:29.335114 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podStartSLOduration=2.335092239 podStartE2EDuration="2.335092239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:29.333165091 +0000 UTC m=+1248.304913963" watchObservedRunningTime="2026-11-04 01:18:29.335092239 +0000 UTC m=+1248.306841112" Nov 04 01:18:29.363264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:29.363193 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:30.301321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:30.301279 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:30.301500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:30.301380 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:30.302241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:30.302222 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:30.302306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:30.302257 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:30.304227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:30.304204 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:30.304363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:30.304237 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:31.306717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:31.302244 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:31.306717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:31.302304 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:31.306717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:31.302244 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:31.306717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:31.302390 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:31.306717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:31.303205 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:31.306717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:31.303215 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:31.306717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:31.303248 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:31.306717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:31.303251 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.341193 5051 generic.go:356] "Generic (PLEG): container finished" podID="d4067319-b8e0-4ec9-8356-93c60b083658" containerID="b1350ae90d59a02fae6f1c684aa0b9988173658e9280448e9c8136f6ecfec681" exitCode=0 Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.341280 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" event={"ID":"d4067319-b8e0-4ec9-8356-93c60b083658","Type":"ContainerDied","Data":"b1350ae90d59a02fae6f1c684aa0b9988173658e9280448e9c8136f6ecfec681"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.341743 5051 scope.go:122] "RemoveContainer" containerID="b1350ae90d59a02fae6f1c684aa0b9988173658e9280448e9c8136f6ecfec681" Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.343377 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.343464 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.343514 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.345622 5051 generic.go:356] "Generic (PLEG): container finished" podID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerID="b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab" exitCode=0 Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.345778 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" event={"ID":"859aa7a6-7b2f-4600-b426-7b89f7416ff0","Type":"ContainerDied","Data":"b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.346215 5051 scope.go:122] "RemoveContainer" containerID="b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab" Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.348326 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.348406 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.348459 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.348495 5051 generic.go:356] "Generic (PLEG): container finished" podID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerID="f27228c0ef1e0692373eebc2f18ef46d8f2b15d15eed76bcb07e5a5259018091" exitCode=0 Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.348658 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" event={"ID":"a3170ff2-51dd-4c54-a3ea-8494c86040c9","Type":"ContainerDied","Data":"f27228c0ef1e0692373eebc2f18ef46d8f2b15d15eed76bcb07e5a5259018091"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.348948 5051 scope.go:122] "RemoveContainer" containerID="f27228c0ef1e0692373eebc2f18ef46d8f2b15d15eed76bcb07e5a5259018091" Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.351101 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.351192 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:32.730578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:32.351242 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:33.360182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.360140 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" event={"ID":"a3170ff2-51dd-4c54-a3ea-8494c86040c9","Type":"ContainerStarted","Data":"e1ea6229a2c7fb80f2eb2d426eb6f44ffefd2f6d290bf91d435e6ec3a0e24170"} Nov 04 01:18:33.362145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.362102 5051 generic.go:356] "Generic (PLEG): container finished" podID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerID="b518baeae647a06d1a41d7f38cf8ca6dad71b7d72250c7ab94a71474e0c455c8" exitCode=0 Nov 04 01:18:33.362297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.362232 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" event={"ID":"5d5de76c-942c-4e1c-bd03-367aca840dbd","Type":"ContainerDied","Data":"b518baeae647a06d1a41d7f38cf8ca6dad71b7d72250c7ab94a71474e0c455c8"} Nov 04 01:18:33.362617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.362595 5051 scope.go:122] "RemoveContainer" containerID="b518baeae647a06d1a41d7f38cf8ca6dad71b7d72250c7ab94a71474e0c455c8" Nov 04 01:18:33.364761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.364724 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:33.364863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.364818 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:33.364928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.364873 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:33.366157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.366102 5051 generic.go:356] "Generic (PLEG): container finished" podID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerID="96c6c0a31fb77cfe64fe167823a1e8511fe0d916b6e4f81505c345905878c5e6" exitCode=0 Nov 04 01:18:33.366338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.366223 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" event={"ID":"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6","Type":"ContainerDied","Data":"96c6c0a31fb77cfe64fe167823a1e8511fe0d916b6e4f81505c345905878c5e6"} Nov 04 01:18:33.366595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.366575 5051 scope.go:122] "RemoveContainer" containerID="96c6c0a31fb77cfe64fe167823a1e8511fe0d916b6e4f81505c345905878c5e6" Nov 04 01:18:33.369793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.369762 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:33.369932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.369848 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:33.369932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.369898 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:33.372043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.371976 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" event={"ID":"d4067319-b8e0-4ec9-8356-93c60b083658","Type":"ContainerStarted","Data":"8c5d9ffb6a57b65b25891a589c2aad928800e76850445ec8af9b3a9d7f3dce3d"} Nov 04 01:18:33.374799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.374777 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" event={"ID":"859aa7a6-7b2f-4600-b426-7b89f7416ff0","Type":"ContainerStarted","Data":"491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450"} Nov 04 01:18:33.539815 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.490597 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:18:33.539815 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.491441 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:33.539815 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.491503 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:33.836685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.836556 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:18:33.903185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.895289 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:33.903185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:33.895377 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:34.205216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.160005 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c"] Nov 04 01:18:34.218140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.216514 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.258743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.228737 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"openshift-service-ca.crt\"" Nov 04 01:18:34.258743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.229491 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"simulator-w5-9c8bf5-kserve-self-signed-certs\"" Nov 04 01:18:34.258743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.229600 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"default-dockercfg-q6h84\"" Nov 04 01:18:34.258743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.229665 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9c8bf5\"/\"kube-root-ca.crt\"" Nov 04 01:18:34.258743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.256360 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:18:34.286432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.281764 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-dshm\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.286432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.281848 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-home\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.286432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.281879 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152bf905-6686-4d5a-aa60-398bd4125980-tls-certs\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.286432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.281961 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9gtc\" (UniqueName: \"kubernetes.io/projected/152bf905-6686-4d5a-aa60-398bd4125980-kube-api-access-j9gtc\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.286432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.282056 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-tmp-dir\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.286432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.282101 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-model-cache\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.291329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.291280 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:34.312605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.306957 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:34.459165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.458644 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c"] Nov 04 01:18:34.511344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.511307 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9gtc\" (UniqueName: \"kubernetes.io/projected/152bf905-6686-4d5a-aa60-398bd4125980-kube-api-access-j9gtc\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.511742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.511719 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-tmp-dir\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.511937 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.511912 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-model-cache\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.516434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.512083 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-dshm\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.516728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.516702 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-home\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.516863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.516848 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152bf905-6686-4d5a-aa60-398bd4125980-tls-certs\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.528678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.521663 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-home\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.528678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.522258 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-tmp-dir\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.529719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.529677 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-model-cache\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.571200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.557250 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:34.571200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.557313 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:34.572861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.572821 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152bf905-6686-4d5a-aa60-398bd4125980-tls-certs\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.591148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.587851 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9gtc\" (UniqueName: \"kubernetes.io/projected/152bf905-6686-4d5a-aa60-398bd4125980-kube-api-access-j9gtc\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.609146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.608880 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" event={"ID":"5d5de76c-942c-4e1c-bd03-367aca840dbd","Type":"ContainerStarted","Data":"ea62e9a01b9efe612b0f85ffa8491e0147459924506595f0e9a57d0392cf092e"} Nov 04 01:18:34.698145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.695967 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-dshm\") pod \"simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.723471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.723331 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:18:34.823861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.823762 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:34.856002 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:18:34.854855 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:18:34.872417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.867380 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:34.872417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:34.867439 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:35.249148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.080333 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh"] Nov 04 01:18:35.279632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.267242 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:35.279632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.267310 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:35.367559 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.347731 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh"] Nov 04 01:18:35.367559 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.357009 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.470174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.447873 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 01:18:35.470174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.447949 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 01:18:35.472205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.471144 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:35.472205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.471198 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:35.525348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.519246 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7486\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-kube-api-access-v7486\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.525348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.519307 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.525348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.519364 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.525348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.519408 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.525348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.519478 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7701537-4386-4a18-bf86-43d4d08a758e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.525348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.519506 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.525348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.519542 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.624236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.624182 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7486\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-kube-api-access-v7486\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.628108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.627830 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.628108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.627953 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.628108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.628029 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.633222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.629845 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7701537-4386-4a18-bf86-43d4d08a758e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.633222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.629907 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.633222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.633046 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.648525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.648482 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:35.945989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.864611 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kwn4w\"" Nov 04 01:18:35.945989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.864723 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:35.945989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.864765 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:35.945989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.909017 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 01:18:35.945989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.944190 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:35.945989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.944253 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:36.006144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.983275 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:36.007028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:35.989679 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" event={"ID":"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6","Type":"ContainerStarted","Data":"d5d0844cd1e594173bb0e0ec07af3ef7297c79d366db28537947136a87db7935"} Nov 04 01:18:36.013505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.011711 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c"] Nov 04 01:18:36.079764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.062876 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7486\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-kube-api-access-v7486\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:36.199659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.199495 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7701537-4386-4a18-bf86-43d4d08a758e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:36.200324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.200272 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:36.233184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.227900 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:36.233184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.231730 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mtcxh\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:36.265147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.239827 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:36.265147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.239882 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:36.391316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.391174 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:18:36.391998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.391666 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:36.391998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.391719 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:36.503317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.502972 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:36.509337 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:18:36.509290 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod152bf905_6686_4d5a_aa60_398bd4125980.slice/crio-f6a8354f58910b55b1e7e1f44c9a1455725eb6ecb605d96fd6b626b414866ec9\": RecentStats: unable to find data in memory cache]" Nov 04 01:18:36.647348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.647309 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:36.647616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.647587 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:36.744314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.727616 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:36.744314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.727700 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:36.801736 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.801698 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" event={"ID":"152bf905-6686-4d5a-aa60-398bd4125980","Type":"ContainerStarted","Data":"f6a8354f58910b55b1e7e1f44c9a1455725eb6ecb605d96fd6b626b414866ec9"} Nov 04 01:18:36.845146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.841229 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:36.845637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:36.845598 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:37.244821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.244787 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:37.245418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.245387 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:37.420193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.420154 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:37.436241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.429180 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:37.511187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.508428 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:37.511421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.511216 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:37.835184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.814727 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:37.863258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.863214 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:37.864054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.836932 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:37.864281 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.864256 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:37.996316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:37.996264 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh"] Nov 04 01:18:38.255159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.244332 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:38.255159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.244405 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:38.399200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.378508 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:38.426636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.426214 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:38.426636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.426273 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:38.480144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.478612 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:38.480144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.478669 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:38.754194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.730678 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:38.754194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.730740 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:38.839928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.839476 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:38.839928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.839541 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:38.968434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.967424 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" event={"ID":"152bf905-6686-4d5a-aa60-398bd4125980","Type":"ContainerStarted","Data":"f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8"} Nov 04 01:18:38.989372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:38.989338 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" event={"ID":"e7701537-4386-4a18-bf86-43d4d08a758e","Type":"ContainerStarted","Data":"4974b88e100aea3c3a6f7d5ad33499633cb905d18f5aeae6f7c3c0b28003a883"} Nov 04 01:18:39.238585 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.238550 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:39.327161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.276419 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:39.381638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.381235 5051 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:39.391205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.386941 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:39.391205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.386995 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:39.467751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.466522 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:39.467751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.466587 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:39.726195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.725927 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:39.726195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.726002 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:39.835145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.834504 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:39.874141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.873470 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:39.874141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:39.873528 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:40.250957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.250508 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:40.250957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.250574 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:40.587204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.530090 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:40.602067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.602026 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:40.606363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.549236 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:40.606522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.606496 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:40.743144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.727479 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:40.743144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.727546 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:40.836142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.835402 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:40.836142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:40.835459 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:41.253398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.253326 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:41.253398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.253385 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:41.418144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.409301 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:41.418144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.409367 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:41.630796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.630697 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:41.631442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.631408 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:41.730798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.730765 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:41.731096 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.731047 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:41.855531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.848833 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 04 01:18:41.855531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:41.848897 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 04 01:18:42.255144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.254221 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:42.255144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.254310 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:42.407145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.398580 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:42.407145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.398640 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:42.470145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.465520 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 04 01:18:42.470145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.465578 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 04 01:18:42.732752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.731929 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:42.732752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.731989 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:42.967163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.960736 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:18:42.967163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:42.962252 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:18:43.013331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:43.013249 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:18:43.075240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:43.066828 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podStartSLOduration=9.066805339 podStartE2EDuration="9.066805339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:39.01753964 +0000 UTC m=+1257.989288515" watchObservedRunningTime="2026-11-04 01:18:43.066805339 +0000 UTC m=+1262.038554210" Nov 04 01:18:43.275782 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:43.275239 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 01:18:43.275782 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:43.275308 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 01:18:43.391178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:43.390696 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:43.391178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:43.390759 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:43.721104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:43.720597 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 04 01:18:43.721104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:43.720660 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 04 01:18:44.254132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.252639 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:18:44.254132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.254030 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:18:44.266160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.264698 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:18:44.272581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.272523 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:18:44.299704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.298404 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:18:44.299704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.299546 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:18:44.321852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.321347 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:18:44.385458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.384813 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 04 01:18:44.385458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.384865 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 04 01:18:44.731633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.729984 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:18:44.731633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.731397 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:18:44.733780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.733758 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:18:44.824625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.824593 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:44.850362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:44.850333 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:45.308276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:45.308227 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:18:45.308785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:45.308301 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:18:45.486721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:45.486668 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:18:45.854049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:45.490436 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:18:45.854049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:45.499976 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:18:46.051580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.049772 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:47414->10.132.0.61:15021: read: connection reset by peer" start-of-body= Nov 04 01:18:46.051580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.049829 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:47414->10.132.0.61:15021: read: connection reset by peer" Nov 04 01:18:46.500172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.499807 5051 generic.go:356] "Generic (PLEG): container finished" podID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerID="9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e" exitCode=0 Nov 04 01:18:46.500172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.499943 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" event={"ID":"9c27e987-5d66-42a9-8ac3-3a377697bc18","Type":"ContainerDied","Data":"9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e"} Nov 04 01:18:46.500719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.500424 5051 scope.go:122] "RemoveContainer" containerID="9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e" Nov 04 01:18:46.502247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.502200 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:46.502377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.502297 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:46.502377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.502347 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:46.504518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.504493 5051 generic.go:356] "Generic (PLEG): container finished" podID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerID="1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c" exitCode=0 Nov 04 01:18:46.504635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.504555 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" event={"ID":"35fd3792-4e02-4f79-9049-8d4bd4973d48","Type":"ContainerDied","Data":"1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c"} Nov 04 01:18:46.505789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.505524 5051 scope.go:122] "RemoveContainer" containerID="1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c" Nov 04 01:18:46.507247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.506861 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:46.507247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.506923 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:46.507247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:46.506954 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:47.522152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:47.522099 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" event={"ID":"9c27e987-5d66-42a9-8ac3-3a377697bc18","Type":"ContainerStarted","Data":"2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9"} Nov 04 01:18:47.533518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:47.533477 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" event={"ID":"35fd3792-4e02-4f79-9049-8d4bd4973d48","Type":"ContainerStarted","Data":"c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487"} Nov 04 01:18:47.928202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:47.926904 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:18:47.928202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:47.927011 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:18:47.928202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:47.927070 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:18:48.074687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:48.074600 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:18:48.075971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:48.074968 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:18:48.075971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:48.075034 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 04 01:18:48.428052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:48.428015 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:48.446311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:48.446277 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:18:48.929908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:48.929442 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:18:48.929908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:48.929504 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:18:49.080183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:49.076489 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:18:49.080183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:49.076551 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 04 01:18:49.912359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:49.911981 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:49.912359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:49.912045 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:18:49.929517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:49.929482 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:18:49.929643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:49.929535 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:18:50.079867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:50.079225 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:18:50.079867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:50.079283 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 04 01:18:50.624147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:50.623629 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" event={"ID":"e7701537-4386-4a18-bf86-43d4d08a758e","Type":"ContainerStarted","Data":"6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b"} Nov 04 01:18:50.644097 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:50.628191 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:50.930590 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:50.930501 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:18:50.930857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:50.930826 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:18:51.076494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:51.076459 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:18:51.076791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:51.076765 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 04 01:18:51.631687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:51.631651 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:18:51.653930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:51.653870 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" podStartSLOduration=4.785291848 podStartE2EDuration="16.653851434s" totalImagesPullingTime="11.868559586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:18:50.67514781 +0000 UTC m=+1269.646896683" watchObservedRunningTime="2026-11-04 01:18:51.653851434 +0000 UTC m=+1270.625600308" Nov 04 01:18:51.937513 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:51.936848 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:18:51.937513 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:51.936914 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:18:52.100013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:52.099899 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:18:52.100013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:52.099974 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 04 01:18:52.926447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:52.926403 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 04 01:18:52.926854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:52.926477 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 04 01:18:53.072550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:53.072512 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 04 01:18:53.072749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:53.072583 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 04 01:18:53.931036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:53.930890 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:18:53.931586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:53.931565 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:18:53.932396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:53.932367 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:18:54.104143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:54.084445 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:18:54.104143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:54.088580 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:18:54.104143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:18:54.091463 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:19:04.106013 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:19:04.105980 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:19:34.152197 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:19:34.151479 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:19:50.973280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:50.973246 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6"] Nov 04 01:19:50.979549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:50.979526 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:50.985568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:50.985545 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:19:50.985728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:50.985591 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6"] Nov 04 01:19:50.985846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:50.985822 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:19:50.985846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:50.985653 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:19:50.985846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:50.985787 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:19:51.102747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.102711 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65ll\" (UniqueName: \"kubernetes.io/projected/054f3b94-1e68-49d0-b304-58c17757c665-kube-api-access-f65ll\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.102971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.102950 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.103049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.102985 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.204300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.204270 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.204300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.204304 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.204546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.204371 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65ll\" (UniqueName: \"kubernetes.io/projected/054f3b94-1e68-49d0-b304-58c17757c665-kube-api-access-f65ll\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.204983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.204959 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.206700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.206678 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.213444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.213420 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65ll\" (UniqueName: \"kubernetes.io/projected/054f3b94-1e68-49d0-b304-58c17757c665-kube-api-access-f65ll\") pod \"maas-api-revoke-sub-bd855fa7-bcrc6\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.301680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.301640 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:19:51.476926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.476896 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz"] Nov 04 01:19:51.481349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.481325 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.489967 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.489941 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz"] Nov 04 01:19:51.590218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.590109 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6"] Nov 04 01:19:51.598245 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:19:51.598217 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054f3b94_1e68_49d0_b304_58c17757c665.slice/crio-5a790d4c83654f611406a19b4fa91624ea0a57100561d738ecc5a364357bcb57 WatchSource:0}: Error finding container 5a790d4c83654f611406a19b4fa91624ea0a57100561d738ecc5a364357bcb57: Status 404 returned error can't find the container with id 5a790d4c83654f611406a19b4fa91624ea0a57100561d738ecc5a364357bcb57 Nov 04 01:19:51.599976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.599956 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:19:51.607990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.607963 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.608090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.608015 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4nc8\" (UniqueName: \"kubernetes.io/projected/f27c1c45-93f0-4eb5-95a3-890e4549f600-kube-api-access-j4nc8\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.608236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.608214 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.709869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.709826 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.710021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.709922 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.710021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.709969 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4nc8\" (UniqueName: \"kubernetes.io/projected/f27c1c45-93f0-4eb5-95a3-890e4549f600-kube-api-access-j4nc8\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.710674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.710638 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.712360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.712336 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.719025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.718996 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4nc8\" (UniqueName: \"kubernetes.io/projected/f27c1c45-93f0-4eb5-95a3-890e4549f600-kube-api-access-j4nc8\") pod \"maas-api-revoke-sub-01b495d9-5wrrz\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:51.811382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:51.811111 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:19:52.279107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.279070 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" event={"ID":"054f3b94-1e68-49d0-b304-58c17757c665","Type":"ContainerStarted","Data":"5a790d4c83654f611406a19b4fa91624ea0a57100561d738ecc5a364357bcb57"} Nov 04 01:19:52.365051 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.365015 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb"] Nov 04 01:19:52.370234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.370213 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.375478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.375452 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb"] Nov 04 01:19:52.478789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.478748 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.478957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.478937 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.479011 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.478969 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdqf\" (UniqueName: \"kubernetes.io/projected/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-kube-api-access-sfdqf\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.494852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.494821 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz"] Nov 04 01:19:52.579985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.579901 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.580151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.580070 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.580151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.580099 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdqf\" (UniqueName: \"kubernetes.io/projected/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-kube-api-access-sfdqf\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.580848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.580821 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.583377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.583352 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.588368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.588339 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdqf\" (UniqueName: \"kubernetes.io/projected/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-kube-api-access-sfdqf\") pod \"maas-api-revoke-sub-0e91b975-dhblb\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.691354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.691322 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:19:52.992050 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:19:52.992010 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c314ef_9509_4dd6_8e3d_ceedf1dcf5ee.slice/crio-75f4086823ca1f42e41d727e67f048bb657a5dfdb37c6022a177023690430a3d WatchSource:0}: Error finding container 75f4086823ca1f42e41d727e67f048bb657a5dfdb37c6022a177023690430a3d: Status 404 returned error can't find the container with id 75f4086823ca1f42e41d727e67f048bb657a5dfdb37c6022a177023690430a3d Nov 04 01:19:52.992814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:52.992784 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb"] Nov 04 01:19:53.298331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:53.298159 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" event={"ID":"f27c1c45-93f0-4eb5-95a3-890e4549f600","Type":"ContainerStarted","Data":"7682765562db966bc4fb23863b7237f26ffb7e37deb400b8450e0baffcae4fc8"} Nov 04 01:19:53.300625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:53.300598 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" event={"ID":"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee","Type":"ContainerStarted","Data":"75f4086823ca1f42e41d727e67f048bb657a5dfdb37c6022a177023690430a3d"} Nov 04 01:19:56.046931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.045143 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq"] Nov 04 01:19:56.061163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.052412 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.061163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.059494 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq"] Nov 04 01:19:56.179954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.178937 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.179954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.179047 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/5211f4db-a6b4-4e7e-9447-010f5b7282d7-kube-api-access-j776s\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.179954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.179102 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.280231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.280193 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.280445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.280350 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/5211f4db-a6b4-4e7e-9447-010f5b7282d7-kube-api-access-j776s\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.280555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.280462 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.280990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.280964 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.285218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.285194 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.296044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.296008 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/5211f4db-a6b4-4e7e-9447-010f5b7282d7-kube-api-access-j776s\") pod \"maas-api-revoke-sub-e9d0485d-dwtlq\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.385930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.385827 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:19:56.766883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:56.766802 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq"] Nov 04 01:19:57.362463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:57.362424 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" event={"ID":"5211f4db-a6b4-4e7e-9447-010f5b7282d7","Type":"ContainerStarted","Data":"ebc09b65d94f9961db97e2c513f64892b92e9ad6a0570fef995a90b70fc52b7a"} Nov 04 01:19:58.235168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.233579 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87"] Nov 04 01:19:58.243911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.243885 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.244808 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.244782 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87"] Nov 04 01:19:58.318087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.318059 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.318376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.318356 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bppvg\" (UniqueName: \"kubernetes.io/projected/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-kube-api-access-bppvg\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.318457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.318419 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.419674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.419639 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bppvg\" (UniqueName: \"kubernetes.io/projected/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-kube-api-access-bppvg\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.420089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.419684 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.420089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.419764 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.420552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.420531 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.423045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.423021 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.430865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.429334 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bppvg\" (UniqueName: \"kubernetes.io/projected/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-kube-api-access-bppvg\") pod \"maas-api-revoke-sub-522fee64-cgz87\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.579331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.579307 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:19:58.887555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:58.887527 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87"] Nov 04 01:19:58.891661 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:19:58.891634 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e9077a_c19b_407d_9b95_c8b6a9bfa2b7.slice/crio-bc3fc72d78a8123e2886e999f67c909062ea288a775f67a42ce14c52c6d1a33d WatchSource:0}: Error finding container bc3fc72d78a8123e2886e999f67c909062ea288a775f67a42ce14c52c6d1a33d: Status 404 returned error can't find the container with id bc3fc72d78a8123e2886e999f67c909062ea288a775f67a42ce14c52c6d1a33d Nov 04 01:19:59.384160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.384105 5051 generic.go:356] "Generic (PLEG): container finished" podID="10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee" containerID="d841f907659d2433866e17c40d6861655f67cf5ddc8eb5b808583121f10d5a02" exitCode=0 Nov 04 01:19:59.384160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.384156 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" event={"ID":"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee","Type":"ContainerDied","Data":"d841f907659d2433866e17c40d6861655f67cf5ddc8eb5b808583121f10d5a02"} Nov 04 01:19:59.385928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.385898 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" event={"ID":"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7","Type":"ContainerStarted","Data":"9f0a9ad3489c637b2bb6fa948450f165473325afa233bfd43ed05fcbf95b9ebb"} Nov 04 01:19:59.386081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.385951 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" event={"ID":"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7","Type":"ContainerStarted","Data":"bc3fc72d78a8123e2886e999f67c909062ea288a775f67a42ce14c52c6d1a33d"} Nov 04 01:19:59.387645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.387615 5051 generic.go:356] "Generic (PLEG): container finished" podID="f27c1c45-93f0-4eb5-95a3-890e4549f600" containerID="4227997806ec4d0f397fc60800b40fc1e643629da9e41b9391bee033dcf1769f" exitCode=0 Nov 04 01:19:59.387765 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.387701 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" event={"ID":"f27c1c45-93f0-4eb5-95a3-890e4549f600","Type":"ContainerDied","Data":"4227997806ec4d0f397fc60800b40fc1e643629da9e41b9391bee033dcf1769f"} Nov 04 01:19:59.389211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.389183 5051 generic.go:356] "Generic (PLEG): container finished" podID="054f3b94-1e68-49d0-b304-58c17757c665" containerID="7b9cab002a3e94a5a5f60c1da764dfdb270d7de998f5cddde3614bc2e39b0524" exitCode=0 Nov 04 01:19:59.389320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.389218 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" event={"ID":"054f3b94-1e68-49d0-b304-58c17757c665","Type":"ContainerDied","Data":"7b9cab002a3e94a5a5f60c1da764dfdb270d7de998f5cddde3614bc2e39b0524"} Nov 04 01:19:59.453284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:19:59.453198 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" podStartSLOduration=1.4531836980000001 podStartE2EDuration="1.453183698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:19:59.450097299 +0000 UTC m=+1338.421846171" watchObservedRunningTime="2026-11-04 01:19:59.453183698 +0000 UTC m=+1338.424932569" Nov 04 01:20:00.399045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.399012 5051 generic.go:356] "Generic (PLEG): container finished" podID="5211f4db-a6b4-4e7e-9447-010f5b7282d7" containerID="0c2635431de4b23f0c604868916a762bd1e32090e960496d5869b9bcade94e2e" exitCode=0 Nov 04 01:20:00.399211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.399096 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" event={"ID":"5211f4db-a6b4-4e7e-9447-010f5b7282d7","Type":"ContainerDied","Data":"0c2635431de4b23f0c604868916a762bd1e32090e960496d5869b9bcade94e2e"} Nov 04 01:20:00.400962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.400896 5051 generic.go:356] "Generic (PLEG): container finished" podID="30e9077a-c19b-407d-9b95-c8b6a9bfa2b7" containerID="9f0a9ad3489c637b2bb6fa948450f165473325afa233bfd43ed05fcbf95b9ebb" exitCode=0 Nov 04 01:20:00.400962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.400945 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" event={"ID":"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7","Type":"ContainerDied","Data":"9f0a9ad3489c637b2bb6fa948450f165473325afa233bfd43ed05fcbf95b9ebb"} Nov 04 01:20:00.606783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.606755 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:20:00.684841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.684814 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:20:00.689403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.689372 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.757628 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-openshift-ca\") pod \"054f3b94-1e68-49d0-b304-58c17757c665\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.757686 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-xks-ca\") pod \"054f3b94-1e68-49d0-b304-58c17757c665\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.757743 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-xks-ca\") pod \"f27c1c45-93f0-4eb5-95a3-890e4549f600\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.757793 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-openshift-ca\") pod \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.757866 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-kube-api-access-sfdqf\" (UniqueName: \"kubernetes.io/projected/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-kube-api-access-sfdqf\") pod \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.757926 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054f3b94-1e68-49d0-b304-58c17757c665-kube-api-access-f65ll\" (UniqueName: \"kubernetes.io/projected/054f3b94-1e68-49d0-b304-58c17757c665-kube-api-access-f65ll\") pod \"054f3b94-1e68-49d0-b304-58c17757c665\" (UID: \"054f3b94-1e68-49d0-b304-58c17757c665\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.758045 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27c1c45-93f0-4eb5-95a3-890e4549f600-kube-api-access-j4nc8\" (UniqueName: \"kubernetes.io/projected/f27c1c45-93f0-4eb5-95a3-890e4549f600-kube-api-access-j4nc8\") pod \"f27c1c45-93f0-4eb5-95a3-890e4549f600\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.758072 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-xks-ca\") pod \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\" (UID: \"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.758148 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-openshift-ca\") pod \"f27c1c45-93f0-4eb5-95a3-890e4549f600\" (UID: \"f27c1c45-93f0-4eb5-95a3-890e4549f600\") " Nov 04 01:20:00.758854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.758622 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-openshift-ca" pod "054f3b94-1e68-49d0-b304-58c17757c665" (UID: "054f3b94-1e68-49d0-b304-58c17757c665"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:00.759987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.759702 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-openshift-ca" pod "f27c1c45-93f0-4eb5-95a3-890e4549f600" (UID: "f27c1c45-93f0-4eb5-95a3-890e4549f600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:00.760373 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.760350 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-openshift-ca" pod "10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee" (UID: "10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:00.762551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.762504 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-xks-ca" pod "10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee" (UID: "10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:00.764044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.764014 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27c1c45-93f0-4eb5-95a3-890e4549f600-kube-api-access-j4nc8" pod "f27c1c45-93f0-4eb5-95a3-890e4549f600" (UID: "f27c1c45-93f0-4eb5-95a3-890e4549f600"). InnerVolumeSpecName "kube-api-access-j4nc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:00.764197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.764174 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-kube-api-access-sfdqf" pod "10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee" (UID: "10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee"). InnerVolumeSpecName "kube-api-access-sfdqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:00.764899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.764877 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054f3b94-1e68-49d0-b304-58c17757c665-kube-api-access-f65ll" pod "054f3b94-1e68-49d0-b304-58c17757c665" (UID: "054f3b94-1e68-49d0-b304-58c17757c665"). InnerVolumeSpecName "kube-api-access-f65ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:00.765930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.765909 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-xks-ca" pod "054f3b94-1e68-49d0-b304-58c17757c665" (UID: "054f3b94-1e68-49d0-b304-58c17757c665"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:00.766865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.766835 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-xks-ca" pod "f27c1c45-93f0-4eb5-95a3-890e4549f600" (UID: "f27c1c45-93f0-4eb5-95a3-890e4549f600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:00.860014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.859977 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4nc8\" (UniqueName: \"kubernetes.io/projected/f27c1c45-93f0-4eb5-95a3-890e4549f600-kube-api-access-j4nc8\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:00.860014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.860015 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:00.860246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.860030 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:00.860246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.860046 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:00.860246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.860060 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/054f3b94-1e68-49d0-b304-58c17757c665-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:00.860246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.860074 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f27c1c45-93f0-4eb5-95a3-890e4549f600-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:00.860246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.860088 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:00.860246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.860102 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfdqf\" (UniqueName: \"kubernetes.io/projected/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee-kube-api-access-sfdqf\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:00.860246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:00.860116 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f65ll\" (UniqueName: \"kubernetes.io/projected/054f3b94-1e68-49d0-b304-58c17757c665-kube-api-access-f65ll\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:01.421249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.421170 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" Nov 04 01:20:01.421249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.421170 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6" event={"ID":"054f3b94-1e68-49d0-b304-58c17757c665","Type":"ContainerDied","Data":"5a790d4c83654f611406a19b4fa91624ea0a57100561d738ecc5a364357bcb57"} Nov 04 01:20:01.421493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.421299 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a790d4c83654f611406a19b4fa91624ea0a57100561d738ecc5a364357bcb57" Nov 04 01:20:01.423263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.423242 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" Nov 04 01:20:01.423704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.423630 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb" event={"ID":"10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee","Type":"ContainerDied","Data":"75f4086823ca1f42e41d727e67f048bb657a5dfdb37c6022a177023690430a3d"} Nov 04 01:20:01.423704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.423682 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f4086823ca1f42e41d727e67f048bb657a5dfdb37c6022a177023690430a3d" Nov 04 01:20:01.425661 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.425638 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" Nov 04 01:20:01.426145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.426097 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz" event={"ID":"f27c1c45-93f0-4eb5-95a3-890e4549f600","Type":"ContainerDied","Data":"7682765562db966bc4fb23863b7237f26ffb7e37deb400b8450e0baffcae4fc8"} Nov 04 01:20:01.426231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.426156 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7682765562db966bc4fb23863b7237f26ffb7e37deb400b8450e0baffcae4fc8" Nov 04 01:20:01.651743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.651711 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:20:01.654617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.654592 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:20:01.783447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.783420 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-kube-api-access-bppvg\" (UniqueName: \"kubernetes.io/projected/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-kube-api-access-bppvg\") pod \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " Nov 04 01:20:01.783614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.783479 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-xks-ca\") pod \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " Nov 04 01:20:01.783614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.783538 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-openshift-ca\") pod \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " Nov 04 01:20:01.783614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.783575 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-openshift-ca\") pod \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " Nov 04 01:20:01.783813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.783637 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-xks-ca\") pod \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\" (UID: \"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7\") " Nov 04 01:20:01.783813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.783690 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5211f4db-a6b4-4e7e-9447-010f5b7282d7-kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/5211f4db-a6b4-4e7e-9447-010f5b7282d7-kube-api-access-j776s\") pod \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\" (UID: \"5211f4db-a6b4-4e7e-9447-010f5b7282d7\") " Nov 04 01:20:01.784053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.784020 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-openshift-ca" pod "5211f4db-a6b4-4e7e-9447-010f5b7282d7" (UID: "5211f4db-a6b4-4e7e-9447-010f5b7282d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:01.784334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.784309 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-openshift-ca" pod "30e9077a-c19b-407d-9b95-c8b6a9bfa2b7" (UID: "30e9077a-c19b-407d-9b95-c8b6a9bfa2b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:01.784420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.784318 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:01.785928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.785904 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-xks-ca" pod "5211f4db-a6b4-4e7e-9447-010f5b7282d7" (UID: "5211f4db-a6b4-4e7e-9447-010f5b7282d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:01.786558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.786534 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-xks-ca" pod "30e9077a-c19b-407d-9b95-c8b6a9bfa2b7" (UID: "30e9077a-c19b-407d-9b95-c8b6a9bfa2b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:01.786672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.786555 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5211f4db-a6b4-4e7e-9447-010f5b7282d7-kube-api-access-j776s" pod "5211f4db-a6b4-4e7e-9447-010f5b7282d7" (UID: "5211f4db-a6b4-4e7e-9447-010f5b7282d7"). InnerVolumeSpecName "kube-api-access-j776s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:01.786672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.786581 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-kube-api-access-bppvg" pod "30e9077a-c19b-407d-9b95-c8b6a9bfa2b7" (UID: "30e9077a-c19b-407d-9b95-c8b6a9bfa2b7"). InnerVolumeSpecName "kube-api-access-bppvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:01.886075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.886032 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:01.886075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.886076 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/5211f4db-a6b4-4e7e-9447-010f5b7282d7-kube-api-access-j776s\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:01.886293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.886091 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bppvg\" (UniqueName: \"kubernetes.io/projected/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-kube-api-access-bppvg\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:01.886293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.886106 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211f4db-a6b4-4e7e-9447-010f5b7282d7-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:01.886293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:01.886157 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:02.434028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:02.433994 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" Nov 04 01:20:02.434239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:02.434030 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87" event={"ID":"30e9077a-c19b-407d-9b95-c8b6a9bfa2b7","Type":"ContainerDied","Data":"bc3fc72d78a8123e2886e999f67c909062ea288a775f67a42ce14c52c6d1a33d"} Nov 04 01:20:02.434239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:02.434090 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc3fc72d78a8123e2886e999f67c909062ea288a775f67a42ce14c52c6d1a33d" Nov 04 01:20:02.435945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:02.435830 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" Nov 04 01:20:02.435945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:02.435872 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq" event={"ID":"5211f4db-a6b4-4e7e-9447-010f5b7282d7","Type":"ContainerDied","Data":"ebc09b65d94f9961db97e2c513f64892b92e9ad6a0570fef995a90b70fc52b7a"} Nov 04 01:20:02.435945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:02.435920 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebc09b65d94f9961db97e2c513f64892b92e9ad6a0570fef995a90b70fc52b7a" Nov 04 01:20:04.184140 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:20:04.180445 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:20:06.035255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035223 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k"] Nov 04 01:20:06.035742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035727 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f27c1c45-93f0-4eb5-95a3-890e4549f600" containerName="revoke-keys" Nov 04 01:20:06.035796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035744 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27c1c45-93f0-4eb5-95a3-890e4549f600" containerName="revoke-keys" Nov 04 01:20:06.035796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035757 5051 cpu_manager.go:414] "containerMap: removing container" podUID="054f3b94-1e68-49d0-b304-58c17757c665" containerName="revoke-keys" Nov 04 01:20:06.035796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035762 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054f3b94-1e68-49d0-b304-58c17757c665" containerName="revoke-keys" Nov 04 01:20:06.035796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035769 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5211f4db-a6b4-4e7e-9447-010f5b7282d7" containerName="revoke-keys" Nov 04 01:20:06.035796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035781 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5211f4db-a6b4-4e7e-9447-010f5b7282d7" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035803 5051 cpu_manager.go:414] "containerMap: removing container" podUID="30e9077a-c19b-407d-9b95-c8b6a9bfa2b7" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035808 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e9077a-c19b-407d-9b95-c8b6a9bfa2b7" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035822 5051 cpu_manager.go:414] "containerMap: removing container" podUID="10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035827 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035912 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27c1c45-93f0-4eb5-95a3-890e4549f600" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035919 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="054f3b94-1e68-49d0-b304-58c17757c665" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035925 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5211f4db-a6b4-4e7e-9447-010f5b7282d7" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035935 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e9077a-c19b-407d-9b95-c8b6a9bfa2b7" containerName="revoke-keys" Nov 04 01:20:06.035949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.035942 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee" containerName="revoke-keys" Nov 04 01:20:06.040597 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.040582 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.045556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.045534 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:20:06.045736 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.045585 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:20:06.045838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.045633 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:06.045945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.045699 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:20:06.048610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.048592 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k"] Nov 04 01:20:06.155303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.155268 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7f8w\" (UniqueName: \"kubernetes.io/projected/a97950fd-de4b-4166-8555-a4eba0ebaf0d-kube-api-access-d7f8w\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.155481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.155327 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.155481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.155461 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.256441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.256408 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.256646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.256532 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7f8w\" (UniqueName: \"kubernetes.io/projected/a97950fd-de4b-4166-8555-a4eba0ebaf0d-kube-api-access-d7f8w\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.256646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.256555 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.257384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.257354 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.258969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.258945 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.266289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.266263 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7f8w\" (UniqueName: \"kubernetes.io/projected/a97950fd-de4b-4166-8555-a4eba0ebaf0d-kube-api-access-d7f8w\") pod \"maas-api-revoke-sub-dd365704-mbf6k\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.374319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.374237 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:06.658702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.658670 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k"] Nov 04 01:20:06.659405 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:06.659360 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97950fd_de4b_4166_8555_a4eba0ebaf0d.slice/crio-23dbe7428ceffe25542149c1a39a9a0fb3fae35d0b54a54d23f0b565c889c9bf WatchSource:0}: Error finding container 23dbe7428ceffe25542149c1a39a9a0fb3fae35d0b54a54d23f0b565c889c9bf: Status 404 returned error can't find the container with id 23dbe7428ceffe25542149c1a39a9a0fb3fae35d0b54a54d23f0b565c889c9bf Nov 04 01:20:06.827788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.827748 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5"] Nov 04 01:20:06.833028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.833006 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:06.840589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.840560 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5"] Nov 04 01:20:06.963540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.963512 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:06.963804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.963768 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:06.963912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:06.963897 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2g5\" (UniqueName: \"kubernetes.io/projected/7158263f-6f8d-4f6c-835b-36899206e24d-kube-api-access-9r2g5\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:07.065497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.065463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:07.065885 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.065557 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2g5\" (UniqueName: \"kubernetes.io/projected/7158263f-6f8d-4f6c-835b-36899206e24d-kube-api-access-9r2g5\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:07.065885 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.065591 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:07.066205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.066179 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:07.068045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.068026 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:07.074928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.074909 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2g5\" (UniqueName: \"kubernetes.io/projected/7158263f-6f8d-4f6c-835b-36899206e24d-kube-api-access-9r2g5\") pod \"maas-api-revoke-sub-5d113800-qlsp5\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:07.149337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.149289 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:07.433565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.433536 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5"] Nov 04 01:20:07.435527 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:07.435488 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7158263f_6f8d_4f6c_835b_36899206e24d.slice/crio-552fcfbf45893c996b1b91a7bee55efd9f56f254f22a767bb86aa5fa4b2c3219 WatchSource:0}: Error finding container 552fcfbf45893c996b1b91a7bee55efd9f56f254f22a767bb86aa5fa4b2c3219: Status 404 returned error can't find the container with id 552fcfbf45893c996b1b91a7bee55efd9f56f254f22a767bb86aa5fa4b2c3219 Nov 04 01:20:07.476540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.476476 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" event={"ID":"a97950fd-de4b-4166-8555-a4eba0ebaf0d","Type":"ContainerStarted","Data":"b4d124ea5b0ce52fd1c91b3f8cd4346febbeb5bfa75bb22b94603f038eb09ec3"} Nov 04 01:20:07.476540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.476526 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" event={"ID":"a97950fd-de4b-4166-8555-a4eba0ebaf0d","Type":"ContainerStarted","Data":"23dbe7428ceffe25542149c1a39a9a0fb3fae35d0b54a54d23f0b565c889c9bf"} Nov 04 01:20:07.477747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.477721 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" event={"ID":"7158263f-6f8d-4f6c-835b-36899206e24d","Type":"ContainerStarted","Data":"552fcfbf45893c996b1b91a7bee55efd9f56f254f22a767bb86aa5fa4b2c3219"} Nov 04 01:20:07.500430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:07.500379 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" podStartSLOduration=1.500365019 podStartE2EDuration="1.500365019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:07.499741529 +0000 UTC m=+1346.471490401" watchObservedRunningTime="2026-11-04 01:20:07.500365019 +0000 UTC m=+1346.472113891" Nov 04 01:20:08.486228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:08.486181 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" event={"ID":"7158263f-6f8d-4f6c-835b-36899206e24d","Type":"ContainerStarted","Data":"9c62173e4d00315b43ca73eeeb8df5f526a4e734090ffb48a97f68c3df01ebba"} Nov 04 01:20:08.488091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:08.488062 5051 generic.go:356] "Generic (PLEG): container finished" podID="a97950fd-de4b-4166-8555-a4eba0ebaf0d" containerID="b4d124ea5b0ce52fd1c91b3f8cd4346febbeb5bfa75bb22b94603f038eb09ec3" exitCode=0 Nov 04 01:20:08.488245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:08.488172 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" event={"ID":"a97950fd-de4b-4166-8555-a4eba0ebaf0d","Type":"ContainerDied","Data":"b4d124ea5b0ce52fd1c91b3f8cd4346febbeb5bfa75bb22b94603f038eb09ec3"} Nov 04 01:20:08.520434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:08.520395 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" podStartSLOduration=2.520383912 podStartE2EDuration="2.520383912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:08.516954974 +0000 UTC m=+1347.488703845" watchObservedRunningTime="2026-11-04 01:20:08.520383912 +0000 UTC m=+1347.492132783" Nov 04 01:20:09.497592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.497555 5051 generic.go:356] "Generic (PLEG): container finished" podID="7158263f-6f8d-4f6c-835b-36899206e24d" containerID="9c62173e4d00315b43ca73eeeb8df5f526a4e734090ffb48a97f68c3df01ebba" exitCode=0 Nov 04 01:20:09.498054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.497641 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" event={"ID":"7158263f-6f8d-4f6c-835b-36899206e24d","Type":"ContainerDied","Data":"9c62173e4d00315b43ca73eeeb8df5f526a4e734090ffb48a97f68c3df01ebba"} Nov 04 01:20:09.652634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.652612 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:09.798109 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.798038 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97950fd-de4b-4166-8555-a4eba0ebaf0d-kube-api-access-d7f8w\" (UniqueName: \"kubernetes.io/projected/a97950fd-de4b-4166-8555-a4eba0ebaf0d-kube-api-access-d7f8w\") pod \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " Nov 04 01:20:09.798109 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.798097 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-openshift-ca\") pod \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " Nov 04 01:20:09.798323 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.798305 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-xks-ca\") pod \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\" (UID: \"a97950fd-de4b-4166-8555-a4eba0ebaf0d\") " Nov 04 01:20:09.798432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.798405 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-openshift-ca" pod "a97950fd-de4b-4166-8555-a4eba0ebaf0d" (UID: "a97950fd-de4b-4166-8555-a4eba0ebaf0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:09.798698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.798671 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:09.800327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.800298 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-xks-ca" pod "a97950fd-de4b-4166-8555-a4eba0ebaf0d" (UID: "a97950fd-de4b-4166-8555-a4eba0ebaf0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:09.800416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.800331 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97950fd-de4b-4166-8555-a4eba0ebaf0d-kube-api-access-d7f8w" pod "a97950fd-de4b-4166-8555-a4eba0ebaf0d" (UID: "a97950fd-de4b-4166-8555-a4eba0ebaf0d"). InnerVolumeSpecName "kube-api-access-d7f8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:09.899492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.899466 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a97950fd-de4b-4166-8555-a4eba0ebaf0d-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:09.899492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:09.899490 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7f8w\" (UniqueName: \"kubernetes.io/projected/a97950fd-de4b-4166-8555-a4eba0ebaf0d-kube-api-access-d7f8w\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:10.506012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.505981 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" Nov 04 01:20:10.506012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.506002 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k" event={"ID":"a97950fd-de4b-4166-8555-a4eba0ebaf0d","Type":"ContainerDied","Data":"23dbe7428ceffe25542149c1a39a9a0fb3fae35d0b54a54d23f0b565c889c9bf"} Nov 04 01:20:10.506600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.506045 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23dbe7428ceffe25542149c1a39a9a0fb3fae35d0b54a54d23f0b565c889c9bf" Nov 04 01:20:10.706735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.706709 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:10.816603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.816501 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7158263f-6f8d-4f6c-835b-36899206e24d-kube-api-access-9r2g5\" (UniqueName: \"kubernetes.io/projected/7158263f-6f8d-4f6c-835b-36899206e24d-kube-api-access-9r2g5\") pod \"7158263f-6f8d-4f6c-835b-36899206e24d\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " Nov 04 01:20:10.816789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.816614 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-openshift-ca\") pod \"7158263f-6f8d-4f6c-835b-36899206e24d\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " Nov 04 01:20:10.816789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.816707 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-xks-ca\") pod \"7158263f-6f8d-4f6c-835b-36899206e24d\" (UID: \"7158263f-6f8d-4f6c-835b-36899206e24d\") " Nov 04 01:20:10.817350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.817317 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-openshift-ca" pod "7158263f-6f8d-4f6c-835b-36899206e24d" (UID: "7158263f-6f8d-4f6c-835b-36899206e24d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:10.819218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.819177 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7158263f-6f8d-4f6c-835b-36899206e24d-kube-api-access-9r2g5" pod "7158263f-6f8d-4f6c-835b-36899206e24d" (UID: "7158263f-6f8d-4f6c-835b-36899206e24d"). InnerVolumeSpecName "kube-api-access-9r2g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:10.819314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.819223 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-xks-ca" pod "7158263f-6f8d-4f6c-835b-36899206e24d" (UID: "7158263f-6f8d-4f6c-835b-36899206e24d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:10.920005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.919962 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:10.920005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.920004 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2g5\" (UniqueName: \"kubernetes.io/projected/7158263f-6f8d-4f6c-835b-36899206e24d-kube-api-access-9r2g5\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:10.920270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:10.920022 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7158263f-6f8d-4f6c-835b-36899206e24d-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:11.513448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:11.513413 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" Nov 04 01:20:11.513844 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:11.513449 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5" event={"ID":"7158263f-6f8d-4f6c-835b-36899206e24d","Type":"ContainerDied","Data":"552fcfbf45893c996b1b91a7bee55efd9f56f254f22a767bb86aa5fa4b2c3219"} Nov 04 01:20:11.513844 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:11.513495 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="552fcfbf45893c996b1b91a7bee55efd9f56f254f22a767bb86aa5fa4b2c3219" Nov 04 01:20:13.659255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.659220 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt"] Nov 04 01:20:13.659934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.659913 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7158263f-6f8d-4f6c-835b-36899206e24d" containerName="revoke-keys" Nov 04 01:20:13.660061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.659935 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7158263f-6f8d-4f6c-835b-36899206e24d" containerName="revoke-keys" Nov 04 01:20:13.660061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.659983 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a97950fd-de4b-4166-8555-a4eba0ebaf0d" containerName="revoke-keys" Nov 04 01:20:13.660061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.659993 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97950fd-de4b-4166-8555-a4eba0ebaf0d" containerName="revoke-keys" Nov 04 01:20:13.660241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.660134 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7158263f-6f8d-4f6c-835b-36899206e24d" containerName="revoke-keys" Nov 04 01:20:13.660241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.660158 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97950fd-de4b-4166-8555-a4eba0ebaf0d" containerName="revoke-keys" Nov 04 01:20:13.663722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.663699 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.669262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.669110 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:20:13.669262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.669178 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:13.669262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.669257 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:20:13.672229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.672207 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:20:13.690589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.690560 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt"] Nov 04 01:20:13.752541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.752508 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.752849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.752821 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.752978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.752896 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgc9l\" (UniqueName: \"kubernetes.io/projected/fb591b4a-c0f6-448b-974d-023ca9620ecb-kube-api-access-jgc9l\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.853950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.853916 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.854146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.853992 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.854227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.854157 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgc9l\" (UniqueName: \"kubernetes.io/projected/fb591b4a-c0f6-448b-974d-023ca9620ecb-kube-api-access-jgc9l\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.854837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.854809 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.856520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.856495 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.863732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.863705 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgc9l\" (UniqueName: \"kubernetes.io/projected/fb591b4a-c0f6-448b-974d-023ca9620ecb-kube-api-access-jgc9l\") pod \"maas-api-revoke-sub-9207e79f-zgbtt\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:13.986094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:13.986066 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:14.268534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:14.268504 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt"] Nov 04 01:20:14.269223 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:14.269197 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb591b4a_c0f6_448b_974d_023ca9620ecb.slice/crio-9cf3b88daf1b15f23745dbe3cd8307d3217182c40c36461cb767c3adaa861c1b WatchSource:0}: Error finding container 9cf3b88daf1b15f23745dbe3cd8307d3217182c40c36461cb767c3adaa861c1b: Status 404 returned error can't find the container with id 9cf3b88daf1b15f23745dbe3cd8307d3217182c40c36461cb767c3adaa861c1b Nov 04 01:20:14.544296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:14.542378 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" event={"ID":"fb591b4a-c0f6-448b-974d-023ca9620ecb","Type":"ContainerStarted","Data":"9cf3b88daf1b15f23745dbe3cd8307d3217182c40c36461cb767c3adaa861c1b"} Nov 04 01:20:15.551128 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:15.551096 5051 generic.go:356] "Generic (PLEG): container finished" podID="fb591b4a-c0f6-448b-974d-023ca9620ecb" containerID="422d3ab04505d30171637b414afbe5bf51dc9f9d61724ccd47a4cd3a00848c5d" exitCode=0 Nov 04 01:20:15.551495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:15.551195 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" event={"ID":"fb591b4a-c0f6-448b-974d-023ca9620ecb","Type":"ContainerDied","Data":"422d3ab04505d30171637b414afbe5bf51dc9f9d61724ccd47a4cd3a00848c5d"} Nov 04 01:20:17.089554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.089229 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:17.148145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.147947 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-xks-ca\") pod \"fb591b4a-c0f6-448b-974d-023ca9620ecb\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " Nov 04 01:20:17.148334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.148242 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-openshift-ca\") pod \"fb591b4a-c0f6-448b-974d-023ca9620ecb\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " Nov 04 01:20:17.148405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.148352 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb591b4a-c0f6-448b-974d-023ca9620ecb-kube-api-access-jgc9l\" (UniqueName: \"kubernetes.io/projected/fb591b4a-c0f6-448b-974d-023ca9620ecb-kube-api-access-jgc9l\") pod \"fb591b4a-c0f6-448b-974d-023ca9620ecb\" (UID: \"fb591b4a-c0f6-448b-974d-023ca9620ecb\") " Nov 04 01:20:17.174265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.153493 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-openshift-ca" pod "fb591b4a-c0f6-448b-974d-023ca9620ecb" (UID: "fb591b4a-c0f6-448b-974d-023ca9620ecb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:17.174265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.170463 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-xks-ca" pod "fb591b4a-c0f6-448b-974d-023ca9620ecb" (UID: "fb591b4a-c0f6-448b-974d-023ca9620ecb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:17.213176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.204201 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb591b4a-c0f6-448b-974d-023ca9620ecb-kube-api-access-jgc9l" pod "fb591b4a-c0f6-448b-974d-023ca9620ecb" (UID: "fb591b4a-c0f6-448b-974d-023ca9620ecb"). InnerVolumeSpecName "kube-api-access-jgc9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:17.249729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.249676 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:17.249729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.249701 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb591b4a-c0f6-448b-974d-023ca9620ecb-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:17.249729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.249712 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgc9l\" (UniqueName: \"kubernetes.io/projected/fb591b4a-c0f6-448b-974d-023ca9620ecb-kube-api-access-jgc9l\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:17.574023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.573983 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" event={"ID":"fb591b4a-c0f6-448b-974d-023ca9620ecb","Type":"ContainerDied","Data":"9cf3b88daf1b15f23745dbe3cd8307d3217182c40c36461cb767c3adaa861c1b"} Nov 04 01:20:17.574549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.574529 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cf3b88daf1b15f23745dbe3cd8307d3217182c40c36461cb767c3adaa861c1b" Nov 04 01:20:17.574710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:17.574444 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt" Nov 04 01:20:19.336180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:19.334728 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:19.336180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:19.334938 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:19.484143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:19.483200 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:21.566051 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.566010 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv"] Nov 04 01:20:21.566589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.566572 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fb591b4a-c0f6-448b-974d-023ca9620ecb" containerName="revoke-keys" Nov 04 01:20:21.566633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.566592 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb591b4a-c0f6-448b-974d-023ca9620ecb" containerName="revoke-keys" Nov 04 01:20:21.566689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.566680 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb591b4a-c0f6-448b-974d-023ca9620ecb" containerName="revoke-keys" Nov 04 01:20:21.582694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.576369 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.582694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.580409 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:21.582694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.582147 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv"] Nov 04 01:20:21.582694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.582666 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:20:21.583043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.582939 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:20:21.583043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.583007 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:20:21.604113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.604074 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.604113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.604139 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrvq\" (UniqueName: \"kubernetes.io/projected/f2862d1b-f394-4a99-8b7d-1cf92a506326-kube-api-access-zsrvq\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.604365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.604280 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.705611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.705575 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.705800 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.705622 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrvq\" (UniqueName: \"kubernetes.io/projected/f2862d1b-f394-4a99-8b7d-1cf92a506326-kube-api-access-zsrvq\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.705800 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.705664 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.706466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.706441 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.708073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.708049 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.715099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.715077 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrvq\" (UniqueName: \"kubernetes.io/projected/f2862d1b-f394-4a99-8b7d-1cf92a506326-kube-api-access-zsrvq\") pod \"maas-api-revoke-sub-01f7ce3e-nqpjv\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.899981 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.899901 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:21.914834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.914812 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5"] Nov 04 01:20:21.919925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.919904 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:21.928693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:21.928663 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5"] Nov 04 01:20:22.009547 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.009512 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.009713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.009569 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.009713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.009602 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-677v6\" (UniqueName: \"kubernetes.io/projected/5c59d196-d8d8-451c-bd32-61657f80304f-kube-api-access-677v6\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.111105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.111062 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.111105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.111109 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.111323 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.111161 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-677v6\" (UniqueName: \"kubernetes.io/projected/5c59d196-d8d8-451c-bd32-61657f80304f-kube-api-access-677v6\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.111901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.111878 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.113728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.113705 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.119200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.119179 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-677v6\" (UniqueName: \"kubernetes.io/projected/5c59d196-d8d8-451c-bd32-61657f80304f-kube-api-access-677v6\") pod \"maas-api-revoke-sub-e8276ec5-9j2k5\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.184864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.184838 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv"] Nov 04 01:20:22.186577 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:22.186547 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2862d1b_f394_4a99_8b7d_1cf92a506326.slice/crio-1bf85a059c7e88119101a1cce7907b9826aeff6617b317b28b5aeaff4a2a86b3 WatchSource:0}: Error finding container 1bf85a059c7e88119101a1cce7907b9826aeff6617b317b28b5aeaff4a2a86b3: Status 404 returned error can't find the container with id 1bf85a059c7e88119101a1cce7907b9826aeff6617b317b28b5aeaff4a2a86b3 Nov 04 01:20:22.255149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.255113 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:22.594975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.594951 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5"] Nov 04 01:20:22.597051 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:22.597026 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c59d196_d8d8_451c_bd32_61657f80304f.slice/crio-ce88d9d2422e54bec8a1a6246c43aa0a2ad7b261b15df9f437ce6642fe8d4bd7 WatchSource:0}: Error finding container ce88d9d2422e54bec8a1a6246c43aa0a2ad7b261b15df9f437ce6642fe8d4bd7: Status 404 returned error can't find the container with id ce88d9d2422e54bec8a1a6246c43aa0a2ad7b261b15df9f437ce6642fe8d4bd7 Nov 04 01:20:22.660894 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.660853 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" event={"ID":"f2862d1b-f394-4a99-8b7d-1cf92a506326","Type":"ContainerStarted","Data":"5be96dd9cf80e39dde0821555c30c6c613d644b80c49d887ca43b3dc5f3b7aca"} Nov 04 01:20:22.661035 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.660916 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" event={"ID":"f2862d1b-f394-4a99-8b7d-1cf92a506326","Type":"ContainerStarted","Data":"1bf85a059c7e88119101a1cce7907b9826aeff6617b317b28b5aeaff4a2a86b3"} Nov 04 01:20:22.662159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.662133 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" event={"ID":"5c59d196-d8d8-451c-bd32-61657f80304f","Type":"ContainerStarted","Data":"ce88d9d2422e54bec8a1a6246c43aa0a2ad7b261b15df9f437ce6642fe8d4bd7"} Nov 04 01:20:22.678390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:22.678342 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" podStartSLOduration=1.678329323 podStartE2EDuration="1.678329323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:22.677043952 +0000 UTC m=+1361.648792823" watchObservedRunningTime="2026-11-04 01:20:22.678329323 +0000 UTC m=+1361.650078210" Nov 04 01:20:23.329793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.329716 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk"] Nov 04 01:20:23.334376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.334358 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.342630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.342602 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk"] Nov 04 01:20:23.441199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.441170 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.441373 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.441355 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.441422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.441384 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlml\" (UniqueName: \"kubernetes.io/projected/8359168a-65f4-4ba8-972b-608282de3f61-kube-api-access-8zlml\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.542061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.542035 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.542244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.542068 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlml\" (UniqueName: \"kubernetes.io/projected/8359168a-65f4-4ba8-972b-608282de3f61-kube-api-access-8zlml\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.542244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.542147 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.542829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.542809 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.544423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.544395 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.550581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.550554 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlml\" (UniqueName: \"kubernetes.io/projected/8359168a-65f4-4ba8-972b-608282de3f61-kube-api-access-8zlml\") pod \"maas-api-revoke-sub-e36c1537-hdphk\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.652826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.652757 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:23.672963 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.672931 5051 generic.go:356] "Generic (PLEG): container finished" podID="f2862d1b-f394-4a99-8b7d-1cf92a506326" containerID="5be96dd9cf80e39dde0821555c30c6c613d644b80c49d887ca43b3dc5f3b7aca" exitCode=0 Nov 04 01:20:23.673075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.673025 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" event={"ID":"f2862d1b-f394-4a99-8b7d-1cf92a506326","Type":"ContainerDied","Data":"5be96dd9cf80e39dde0821555c30c6c613d644b80c49d887ca43b3dc5f3b7aca"} Nov 04 01:20:23.674812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.674787 5051 generic.go:356] "Generic (PLEG): container finished" podID="5c59d196-d8d8-451c-bd32-61657f80304f" containerID="f56004a0e7a8c3a288cb6ce3c77ef7cc1a53c77b5de6cc60ab6ab9b6de04e8d4" exitCode=0 Nov 04 01:20:23.674908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.674867 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" event={"ID":"5c59d196-d8d8-451c-bd32-61657f80304f","Type":"ContainerDied","Data":"f56004a0e7a8c3a288cb6ce3c77ef7cc1a53c77b5de6cc60ab6ab9b6de04e8d4"} Nov 04 01:20:23.941939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:23.941910 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk"] Nov 04 01:20:23.943007 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:23.942983 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8359168a_65f4_4ba8_972b_608282de3f61.slice/crio-73e66cf5cc4d0dcf4f17eb8d91c6ceece1d75336383324906f2a942ef2364617 WatchSource:0}: Error finding container 73e66cf5cc4d0dcf4f17eb8d91c6ceece1d75336383324906f2a942ef2364617: Status 404 returned error can't find the container with id 73e66cf5cc4d0dcf4f17eb8d91c6ceece1d75336383324906f2a942ef2364617 Nov 04 01:20:24.684771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:24.684680 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" event={"ID":"8359168a-65f4-4ba8-972b-608282de3f61","Type":"ContainerStarted","Data":"cb0c41e104c9016085e708da78bffe0e38c63e0162921aa21e8cb6a5d332d2e3"} Nov 04 01:20:24.684771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:24.684758 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" event={"ID":"8359168a-65f4-4ba8-972b-608282de3f61","Type":"ContainerStarted","Data":"73e66cf5cc4d0dcf4f17eb8d91c6ceece1d75336383324906f2a942ef2364617"} Nov 04 01:20:24.706655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:24.706596 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" podStartSLOduration=1.706576213 podStartE2EDuration="1.706576213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:24.703359069 +0000 UTC m=+1363.675107943" watchObservedRunningTime="2026-11-04 01:20:24.706576213 +0000 UTC m=+1363.678325085" Nov 04 01:20:24.967674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:24.967647 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:24.970897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:24.970878 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:25.063265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.063227 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-xks-ca\") pod \"f2862d1b-f394-4a99-8b7d-1cf92a506326\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " Nov 04 01:20:25.063429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.063283 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-openshift-ca\") pod \"5c59d196-d8d8-451c-bd32-61657f80304f\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " Nov 04 01:20:25.063429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.063372 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c59d196-d8d8-451c-bd32-61657f80304f-kube-api-access-677v6\" (UniqueName: \"kubernetes.io/projected/5c59d196-d8d8-451c-bd32-61657f80304f-kube-api-access-677v6\") pod \"5c59d196-d8d8-451c-bd32-61657f80304f\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " Nov 04 01:20:25.063513 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.063448 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-openshift-ca\") pod \"f2862d1b-f394-4a99-8b7d-1cf92a506326\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " Nov 04 01:20:25.063513 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.063489 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-xks-ca\") pod \"5c59d196-d8d8-451c-bd32-61657f80304f\" (UID: \"5c59d196-d8d8-451c-bd32-61657f80304f\") " Nov 04 01:20:25.063615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.063601 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2862d1b-f394-4a99-8b7d-1cf92a506326-kube-api-access-zsrvq\" (UniqueName: \"kubernetes.io/projected/f2862d1b-f394-4a99-8b7d-1cf92a506326-kube-api-access-zsrvq\") pod \"f2862d1b-f394-4a99-8b7d-1cf92a506326\" (UID: \"f2862d1b-f394-4a99-8b7d-1cf92a506326\") " Nov 04 01:20:25.063783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.063738 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-openshift-ca" pod "5c59d196-d8d8-451c-bd32-61657f80304f" (UID: "5c59d196-d8d8-451c-bd32-61657f80304f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:25.063887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.063812 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-openshift-ca" pod "f2862d1b-f394-4a99-8b7d-1cf92a506326" (UID: "f2862d1b-f394-4a99-8b7d-1cf92a506326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:25.064079 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.064052 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:25.064191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.064078 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:25.065693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.065659 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-xks-ca" pod "5c59d196-d8d8-451c-bd32-61657f80304f" (UID: "5c59d196-d8d8-451c-bd32-61657f80304f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:25.065798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.065746 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2862d1b-f394-4a99-8b7d-1cf92a506326-kube-api-access-zsrvq" pod "f2862d1b-f394-4a99-8b7d-1cf92a506326" (UID: "f2862d1b-f394-4a99-8b7d-1cf92a506326"). InnerVolumeSpecName "kube-api-access-zsrvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:25.065904 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.065886 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c59d196-d8d8-451c-bd32-61657f80304f-kube-api-access-677v6" pod "5c59d196-d8d8-451c-bd32-61657f80304f" (UID: "5c59d196-d8d8-451c-bd32-61657f80304f"). InnerVolumeSpecName "kube-api-access-677v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:25.065951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.065937 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-xks-ca" pod "f2862d1b-f394-4a99-8b7d-1cf92a506326" (UID: "f2862d1b-f394-4a99-8b7d-1cf92a506326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:25.165716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.165687 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2862d1b-f394-4a99-8b7d-1cf92a506326-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:25.165716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.165715 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-677v6\" (UniqueName: \"kubernetes.io/projected/5c59d196-d8d8-451c-bd32-61657f80304f-kube-api-access-677v6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:25.165856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.165726 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c59d196-d8d8-451c-bd32-61657f80304f-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:25.165856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.165735 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsrvq\" (UniqueName: \"kubernetes.io/projected/f2862d1b-f394-4a99-8b7d-1cf92a506326-kube-api-access-zsrvq\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:25.694230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.694191 5051 generic.go:356] "Generic (PLEG): container finished" podID="8359168a-65f4-4ba8-972b-608282de3f61" containerID="cb0c41e104c9016085e708da78bffe0e38c63e0162921aa21e8cb6a5d332d2e3" exitCode=0 Nov 04 01:20:25.694698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.694239 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" event={"ID":"8359168a-65f4-4ba8-972b-608282de3f61","Type":"ContainerDied","Data":"cb0c41e104c9016085e708da78bffe0e38c63e0162921aa21e8cb6a5d332d2e3"} Nov 04 01:20:25.695955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.695933 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" Nov 04 01:20:25.696151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.695952 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv" event={"ID":"f2862d1b-f394-4a99-8b7d-1cf92a506326","Type":"ContainerDied","Data":"1bf85a059c7e88119101a1cce7907b9826aeff6617b317b28b5aeaff4a2a86b3"} Nov 04 01:20:25.696151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.695995 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bf85a059c7e88119101a1cce7907b9826aeff6617b317b28b5aeaff4a2a86b3" Nov 04 01:20:25.697650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.697628 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" event={"ID":"5c59d196-d8d8-451c-bd32-61657f80304f","Type":"ContainerDied","Data":"ce88d9d2422e54bec8a1a6246c43aa0a2ad7b261b15df9f437ce6642fe8d4bd7"} Nov 04 01:20:25.697650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.697648 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5" Nov 04 01:20:25.697835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:25.697660 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce88d9d2422e54bec8a1a6246c43aa0a2ad7b261b15df9f437ce6642fe8d4bd7" Nov 04 01:20:26.479896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.479861 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4"] Nov 04 01:20:26.480400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.480383 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5c59d196-d8d8-451c-bd32-61657f80304f" containerName="revoke-keys" Nov 04 01:20:26.480400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.480399 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c59d196-d8d8-451c-bd32-61657f80304f" containerName="revoke-keys" Nov 04 01:20:26.480574 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.480423 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f2862d1b-f394-4a99-8b7d-1cf92a506326" containerName="revoke-keys" Nov 04 01:20:26.480574 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.480428 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2862d1b-f394-4a99-8b7d-1cf92a506326" containerName="revoke-keys" Nov 04 01:20:26.480574 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.480500 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2862d1b-f394-4a99-8b7d-1cf92a506326" containerName="revoke-keys" Nov 04 01:20:26.480574 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.480513 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c59d196-d8d8-451c-bd32-61657f80304f" containerName="revoke-keys" Nov 04 01:20:26.484228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.484205 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.498353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.498265 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4"] Nov 04 01:20:26.583246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.583210 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.583443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.583271 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.583549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.583521 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslp5\" (UniqueName: \"kubernetes.io/projected/24ebe1b7-5a95-455c-91a7-a49694d43109-kube-api-access-vslp5\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.684700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.684672 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.684888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.684777 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vslp5\" (UniqueName: \"kubernetes.io/projected/24ebe1b7-5a95-455c-91a7-a49694d43109-kube-api-access-vslp5\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.684888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.684839 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.685543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.685517 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.687284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.687259 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.692751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.692719 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslp5\" (UniqueName: \"kubernetes.io/projected/24ebe1b7-5a95-455c-91a7-a49694d43109-kube-api-access-vslp5\") pod \"maas-api-revoke-sub-774d6cd7-xvzq4\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.804978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.804949 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:26.880108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.880084 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:26.979468 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.979410 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57"] Nov 04 01:20:26.980270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.980248 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8359168a-65f4-4ba8-972b-608282de3f61" containerName="revoke-keys" Nov 04 01:20:26.980385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.980273 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8359168a-65f4-4ba8-972b-608282de3f61" containerName="revoke-keys" Nov 04 01:20:26.980435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.980396 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8359168a-65f4-4ba8-972b-608282de3f61" containerName="revoke-keys" Nov 04 01:20:27.015443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.991994 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-xks-ca\") pod \"8359168a-65f4-4ba8-972b-608282de3f61\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " Nov 04 01:20:27.015443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.992078 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-openshift-ca\") pod \"8359168a-65f4-4ba8-972b-608282de3f61\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " Nov 04 01:20:27.015443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.992264 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8359168a-65f4-4ba8-972b-608282de3f61-kube-api-access-8zlml\" (UniqueName: \"kubernetes.io/projected/8359168a-65f4-4ba8-972b-608282de3f61-kube-api-access-8zlml\") pod \"8359168a-65f4-4ba8-972b-608282de3f61\" (UID: \"8359168a-65f4-4ba8-972b-608282de3f61\") " Nov 04 01:20:27.015443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:26.999074 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8359168a-65f4-4ba8-972b-608282de3f61-kube-api-access-8zlml" pod "8359168a-65f4-4ba8-972b-608282de3f61" (UID: "8359168a-65f4-4ba8-972b-608282de3f61"). InnerVolumeSpecName "kube-api-access-8zlml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:27.015443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.001433 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-xks-ca" pod "8359168a-65f4-4ba8-972b-608282de3f61" (UID: "8359168a-65f4-4ba8-972b-608282de3f61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:27.015443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.001746 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-openshift-ca" pod "8359168a-65f4-4ba8-972b-608282de3f61" (UID: "8359168a-65f4-4ba8-972b-608282de3f61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:27.015443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.009363 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57"] Nov 04 01:20:27.015443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.009507 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.093532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.093498 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.093682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.093582 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.093682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.093634 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.093764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.093739 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599da2-18ec-4ac3-8968-e6726cb5d369-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.093873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.093853 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.093908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.093893 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.093999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.093984 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.094074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.094029 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.094074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.094052 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkwq\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-kube-api-access-2fkwq\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.094265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.094250 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlml\" (UniqueName: \"kubernetes.io/projected/8359168a-65f4-4ba8-972b-608282de3f61-kube-api-access-8zlml\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:27.094305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.094273 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:27.094305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.094287 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8359168a-65f4-4ba8-972b-608282de3f61-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:27.099572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.099551 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4"] Nov 04 01:20:27.100848 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:27.100815 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ebe1b7_5a95_455c_91a7_a49694d43109.slice/crio-117beb1fc37be05d6fab45fd9409b03eed156f693516cb85c867e1b55b9be943 WatchSource:0}: Error finding container 117beb1fc37be05d6fab45fd9409b03eed156f693516cb85c867e1b55b9be943: Status 404 returned error can't find the container with id 117beb1fc37be05d6fab45fd9409b03eed156f693516cb85c867e1b55b9be943 Nov 04 01:20:27.199412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199260 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.199412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199324 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.199412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199355 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkwq\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-kube-api-access-2fkwq\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.199755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199447 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.199755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199515 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.199755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199585 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.199755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199625 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599da2-18ec-4ac3-8968-e6726cb5d369-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.199755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199734 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.200065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.199772 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.200814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.200784 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.201136 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.201077 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.201239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.201154 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.201842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.201770 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599da2-18ec-4ac3-8968-e6726cb5d369-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.202075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.202030 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.205424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.205398 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.205640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.205622 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.209178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.209156 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.209483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.209465 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkwq\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-kube-api-access-2fkwq\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.335874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.335814 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:27.565988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.565908 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h"] Nov 04 01:20:27.574470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.574442 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.587399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.587372 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h"] Nov 04 01:20:27.637590 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.637502 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57"] Nov 04 01:20:27.651988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.647135 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:27.651988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.647226 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:27.651988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.647266 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:27.708646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.708624 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.708816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.708794 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.708902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.708833 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.708974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.708919 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.709027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.708987 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72n6d\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-kube-api-access-72n6d\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.709178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.709156 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.709216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.709192 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.709272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.709214 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb08d7e7-695c-4005-8101-58b8a376eb7c-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.709342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.709317 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.714948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.714926 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" event={"ID":"60599da2-18ec-4ac3-8968-e6726cb5d369","Type":"ContainerStarted","Data":"fbce760ccb507aced8cb5ee776d9aa2e60bc3b43a837843d11d31ba2e0f2ea13"} Nov 04 01:20:27.716663 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.716581 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" Nov 04 01:20:27.716663 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.716590 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk" event={"ID":"8359168a-65f4-4ba8-972b-608282de3f61","Type":"ContainerDied","Data":"73e66cf5cc4d0dcf4f17eb8d91c6ceece1d75336383324906f2a942ef2364617"} Nov 04 01:20:27.716663 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.716641 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e66cf5cc4d0dcf4f17eb8d91c6ceece1d75336383324906f2a942ef2364617" Nov 04 01:20:27.719388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.719303 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" event={"ID":"24ebe1b7-5a95-455c-91a7-a49694d43109","Type":"ContainerStarted","Data":"183f04992f9642fd7d2c12c4766698bd3f11ba6b728faf8b9235bd2aca007643"} Nov 04 01:20:27.722333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.722307 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" event={"ID":"24ebe1b7-5a95-455c-91a7-a49694d43109","Type":"ContainerStarted","Data":"117beb1fc37be05d6fab45fd9409b03eed156f693516cb85c867e1b55b9be943"} Nov 04 01:20:27.739756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.739714 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" podStartSLOduration=1.739701194 podStartE2EDuration="1.739701194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:27.738276124 +0000 UTC m=+1366.710024994" watchObservedRunningTime="2026-11-04 01:20:27.739701194 +0000 UTC m=+1366.711450065" Nov 04 01:20:27.810616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.810587 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.811095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.811078 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.811966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.811583 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.812660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.812640 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.812835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.812815 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.813629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.813324 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.814056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.813768 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72n6d\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-kube-api-access-72n6d\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.814299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.813601 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.814299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.813255 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.816238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.814689 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.816238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.814733 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.816238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.814770 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb08d7e7-695c-4005-8101-58b8a376eb7c-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.816238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.815540 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb08d7e7-695c-4005-8101-58b8a376eb7c-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.816238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.815806 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.828810 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.828792 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.830352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.830326 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.834702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.832602 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72n6d\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-kube-api-access-72n6d\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.837710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.837687 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:27.897876 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:27.897831 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:28.026413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.026375 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx"] Nov 04 01:20:28.037592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.035742 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.041619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.041551 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx"] Nov 04 01:20:28.119307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.119239 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.119307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.119300 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.119507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.119492 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2zn\" (UniqueName: \"kubernetes.io/projected/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-kube-api-access-fj2zn\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.191816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.191787 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h"] Nov 04 01:20:28.193537 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:28.193503 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb08d7e7_695c_4005_8101_58b8a376eb7c.slice/crio-533d45946591f2a510ecc12c62d9afc01013b87eb9060c18a242f77521bd446a WatchSource:0}: Error finding container 533d45946591f2a510ecc12c62d9afc01013b87eb9060c18a242f77521bd446a: Status 404 returned error can't find the container with id 533d45946591f2a510ecc12c62d9afc01013b87eb9060c18a242f77521bd446a Nov 04 01:20:28.196740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.196697 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:28.196848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.196786 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:28.196913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.196865 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:28.221082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.221060 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2zn\" (UniqueName: \"kubernetes.io/projected/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-kube-api-access-fj2zn\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.221206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.221156 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.221206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.221198 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.221746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.221725 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.223362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.223344 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.233218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.233194 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2zn\" (UniqueName: \"kubernetes.io/projected/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-kube-api-access-fj2zn\") pod \"maas-api-revoke-sub-d3cf45bd-vtfbx\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.364435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.364397 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:28.729639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.729610 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx"] Nov 04 01:20:28.744104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.743905 5051 generic.go:356] "Generic (PLEG): container finished" podID="24ebe1b7-5a95-455c-91a7-a49694d43109" containerID="183f04992f9642fd7d2c12c4766698bd3f11ba6b728faf8b9235bd2aca007643" exitCode=0 Nov 04 01:20:28.744104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.743985 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" event={"ID":"24ebe1b7-5a95-455c-91a7-a49694d43109","Type":"ContainerDied","Data":"183f04992f9642fd7d2c12c4766698bd3f11ba6b728faf8b9235bd2aca007643"} Nov 04 01:20:28.746055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.745989 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" event={"ID":"eb08d7e7-695c-4005-8101-58b8a376eb7c","Type":"ContainerStarted","Data":"28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30"} Nov 04 01:20:28.746055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.746028 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" event={"ID":"eb08d7e7-695c-4005-8101-58b8a376eb7c","Type":"ContainerStarted","Data":"533d45946591f2a510ecc12c62d9afc01013b87eb9060c18a242f77521bd446a"} Nov 04 01:20:28.748275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.747854 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" event={"ID":"60599da2-18ec-4ac3-8968-e6726cb5d369","Type":"ContainerStarted","Data":"648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc"} Nov 04 01:20:28.833042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.832985 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podStartSLOduration=2.832965982 podStartE2EDuration="2.832965982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:28.831793341 +0000 UTC m=+1367.803542208" watchObservedRunningTime="2026-11-04 01:20:28.832965982 +0000 UTC m=+1367.804714871" Nov 04 01:20:28.912083 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.910732 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podStartSLOduration=1.910710193 podStartE2EDuration="1.910710193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:28.909415073 +0000 UTC m=+1367.881163944" watchObservedRunningTime="2026-11-04 01:20:28.910710193 +0000 UTC m=+1367.882459066" Nov 04 01:20:28.925143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:28.925101 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:29.337226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:29.336685 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:29.337860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:29.337837 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:20:29.337902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:29.337886 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:20:29.848145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:29.835858 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" event={"ID":"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571","Type":"ContainerStarted","Data":"87574b9a5ee91fa05dc391ab3dd7a9c0b6b3668d51dc5ee9e18d5f513460d507"} Nov 04 01:20:29.848145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:29.835918 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" event={"ID":"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571","Type":"ContainerStarted","Data":"fb76ae8d41900039e9bff6f83ee800e70f83bdac0e2d75e01d51347ea5bb9a6a"} Nov 04 01:20:29.907529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:29.907219 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:20:29.907529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:29.907281 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 04 01:20:29.931145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:29.927269 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" podStartSLOduration=1.9272509370000002 podStartE2EDuration="1.927250937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:29.90054415 +0000 UTC m=+1368.872293022" watchObservedRunningTime="2026-11-04 01:20:29.927250937 +0000 UTC m=+1368.898999805" Nov 04 01:20:30.341183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:30.340615 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:20:30.341183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:30.340691 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:20:31.036612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.028243 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:20:31.036612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.028302 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 04 01:20:31.169146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.135947 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:31.169146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.137966 5051 generic.go:356] "Generic (PLEG): container finished" podID="e120ca74-3b6f-4241-a8a7-2ce9fd3d0571" containerID="87574b9a5ee91fa05dc391ab3dd7a9c0b6b3668d51dc5ee9e18d5f513460d507" exitCode=0 Nov 04 01:20:31.169146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.139252 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" event={"ID":"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571","Type":"ContainerDied","Data":"87574b9a5ee91fa05dc391ab3dd7a9c0b6b3668d51dc5ee9e18d5f513460d507"} Nov 04 01:20:31.281683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.275484 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-xks-ca\") pod \"24ebe1b7-5a95-455c-91a7-a49694d43109\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " Nov 04 01:20:31.281683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.275596 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ebe1b7-5a95-455c-91a7-a49694d43109-kube-api-access-vslp5\" (UniqueName: \"kubernetes.io/projected/24ebe1b7-5a95-455c-91a7-a49694d43109-kube-api-access-vslp5\") pod \"24ebe1b7-5a95-455c-91a7-a49694d43109\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " Nov 04 01:20:31.281683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.275768 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-openshift-ca\") pod \"24ebe1b7-5a95-455c-91a7-a49694d43109\" (UID: \"24ebe1b7-5a95-455c-91a7-a49694d43109\") " Nov 04 01:20:31.285430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.285381 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-openshift-ca" pod "24ebe1b7-5a95-455c-91a7-a49694d43109" (UID: "24ebe1b7-5a95-455c-91a7-a49694d43109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:31.409153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.379085 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:31.481719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.481447 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ebe1b7-5a95-455c-91a7-a49694d43109-kube-api-access-vslp5" pod "24ebe1b7-5a95-455c-91a7-a49694d43109" (UID: "24ebe1b7-5a95-455c-91a7-a49694d43109"). InnerVolumeSpecName "kube-api-access-vslp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:31.536199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.535191 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vslp5\" (UniqueName: \"kubernetes.io/projected/24ebe1b7-5a95-455c-91a7-a49694d43109-kube-api-access-vslp5\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:31.591153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.576943 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:20:31.591153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.577004 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:20:31.591153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.584224 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-xks-ca" pod "24ebe1b7-5a95-455c-91a7-a49694d43109" (UID: "24ebe1b7-5a95-455c-91a7-a49694d43109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:31.653514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.653459 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ebe1b7-5a95-455c-91a7-a49694d43109-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:31.917203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.914241 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:20:31.917203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:31.914305 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 04 01:20:32.210225 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.204852 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" Nov 04 01:20:32.234266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.225433 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4" event={"ID":"24ebe1b7-5a95-455c-91a7-a49694d43109","Type":"ContainerDied","Data":"117beb1fc37be05d6fab45fd9409b03eed156f693516cb85c867e1b55b9be943"} Nov 04 01:20:32.234266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.225503 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="117beb1fc37be05d6fab45fd9409b03eed156f693516cb85c867e1b55b9be943" Nov 04 01:20:32.337155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.337033 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:20:32.337155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.337088 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:20:32.701144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.699316 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:32.807145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.806599 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-kube-api-access-fj2zn\" (UniqueName: \"kubernetes.io/projected/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-kube-api-access-fj2zn\") pod \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " Nov 04 01:20:32.807145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.806769 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-openshift-ca\") pod \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " Nov 04 01:20:32.807145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.806892 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-xks-ca\") pod \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\" (UID: \"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571\") " Nov 04 01:20:32.817600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.817292 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-openshift-ca" pod "e120ca74-3b6f-4241-a8a7-2ce9fd3d0571" (UID: "e120ca74-3b6f-4241-a8a7-2ce9fd3d0571"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:32.821594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.821214 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-xks-ca" pod "e120ca74-3b6f-4241-a8a7-2ce9fd3d0571" (UID: "e120ca74-3b6f-4241-a8a7-2ce9fd3d0571"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:32.823523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.823488 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-kube-api-access-fj2zn" pod "e120ca74-3b6f-4241-a8a7-2ce9fd3d0571" (UID: "e120ca74-3b6f-4241-a8a7-2ce9fd3d0571"). InnerVolumeSpecName "kube-api-access-fj2zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:32.899194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.898503 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:20:32.899194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.898566 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 04 01:20:32.915023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.914992 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:32.915240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.915224 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj2zn\" (UniqueName: \"kubernetes.io/projected/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-kube-api-access-fj2zn\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:32.915323 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:32.915314 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:33.235099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.234627 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" Nov 04 01:20:33.235099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.234643 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx" event={"ID":"e120ca74-3b6f-4241-a8a7-2ce9fd3d0571","Type":"ContainerDied","Data":"fb76ae8d41900039e9bff6f83ee800e70f83bdac0e2d75e01d51347ea5bb9a6a"} Nov 04 01:20:33.235099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.234690 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb76ae8d41900039e9bff6f83ee800e70f83bdac0e2d75e01d51347ea5bb9a6a" Nov 04 01:20:33.336624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.336528 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:20:33.336624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.336582 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:20:33.637971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.637938 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc"] Nov 04 01:20:33.638742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.638722 5051 cpu_manager.go:414] "containerMap: removing container" podUID="24ebe1b7-5a95-455c-91a7-a49694d43109" containerName="revoke-keys" Nov 04 01:20:33.638863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.638853 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ebe1b7-5a95-455c-91a7-a49694d43109" containerName="revoke-keys" Nov 04 01:20:33.638920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.638913 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e120ca74-3b6f-4241-a8a7-2ce9fd3d0571" containerName="revoke-keys" Nov 04 01:20:33.638966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.638959 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e120ca74-3b6f-4241-a8a7-2ce9fd3d0571" containerName="revoke-keys" Nov 04 01:20:33.639172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.639162 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ebe1b7-5a95-455c-91a7-a49694d43109" containerName="revoke-keys" Nov 04 01:20:33.639240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.639233 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e120ca74-3b6f-4241-a8a7-2ce9fd3d0571" containerName="revoke-keys" Nov 04 01:20:33.645335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.645314 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.648984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.648942 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:33.653726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.653705 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:20:33.657181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.656551 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:20:33.657314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.656644 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:20:33.664140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.660198 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc"] Nov 04 01:20:33.736009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.735975 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfsv9\" (UniqueName: \"kubernetes.io/projected/e9cf6ade-85bd-4c25-a923-4c041c046941-kube-api-access-hfsv9\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.736237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.736193 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.736237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.736224 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.839980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.839932 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.840197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.839987 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.840197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.840076 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfsv9\" (UniqueName: \"kubernetes.io/projected/e9cf6ade-85bd-4c25-a923-4c041c046941-kube-api-access-hfsv9\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.841440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.841407 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.846161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.845317 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.848154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.848105 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfsv9\" (UniqueName: \"kubernetes.io/projected/e9cf6ade-85bd-4c25-a923-4c041c046941-kube-api-access-hfsv9\") pod \"maas-api-revoke-sub-1cddeac7-8vdhc\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:33.914197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.900487 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:20:33.914197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:33.900549 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 04 01:20:34.025564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:34.025533 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:34.146393 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:20:34.146356 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:20:34.336842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:34.336796 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 04 01:20:34.337439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:34.336858 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 04 01:20:34.440711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:34.440684 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc"] Nov 04 01:20:34.440900 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:34.440830 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cf6ade_85bd_4c25_a923_4c041c046941.slice/crio-e24317b6c1b7313600d65e466fa6f039036288e6dd2b1f15842bb1eea0cc14bc WatchSource:0}: Error finding container e24317b6c1b7313600d65e466fa6f039036288e6dd2b1f15842bb1eea0cc14bc: Status 404 returned error can't find the container with id e24317b6c1b7313600d65e466fa6f039036288e6dd2b1f15842bb1eea0cc14bc Nov 04 01:20:34.898401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:34.898171 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:20:34.898401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:34.898224 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 04 01:20:35.269183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:35.261063 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" event={"ID":"e9cf6ade-85bd-4c25-a923-4c041c046941","Type":"ContainerStarted","Data":"60919e92ced66d2bc32d1015bf395c775030b588c6f9277e2a561588609bc223"} Nov 04 01:20:35.269183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:35.261150 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" event={"ID":"e9cf6ade-85bd-4c25-a923-4c041c046941","Type":"ContainerStarted","Data":"e24317b6c1b7313600d65e466fa6f039036288e6dd2b1f15842bb1eea0cc14bc"} Nov 04 01:20:35.294302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:35.294228 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" podStartSLOduration=2.294194311 podStartE2EDuration="2.294194311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:35.291343226 +0000 UTC m=+1374.263092088" watchObservedRunningTime="2026-11-04 01:20:35.294194311 +0000 UTC m=+1374.265943188" Nov 04 01:20:35.899404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:35.899307 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 04 01:20:35.899404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:35.899364 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 04 01:20:36.294144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.293132 5051 generic.go:356] "Generic (PLEG): container finished" podID="e9cf6ade-85bd-4c25-a923-4c041c046941" containerID="60919e92ced66d2bc32d1015bf395c775030b588c6f9277e2a561588609bc223" exitCode=0 Nov 04 01:20:36.294144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.293734 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" event={"ID":"e9cf6ade-85bd-4c25-a923-4c041c046941","Type":"ContainerDied","Data":"60919e92ced66d2bc32d1015bf395c775030b588c6f9277e2a561588609bc223"} Nov 04 01:20:36.396145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.395482 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:20:36.396145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.395572 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:20:36.534450 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.534421 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:36.534955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.534938 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:36.535682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.535664 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:20:36.613979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.613890 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97"] Nov 04 01:20:36.617906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.617882 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.647971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.647944 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97"] Nov 04 01:20:36.653971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.653938 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.654076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.653990 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.654273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.654244 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4g76\" (UniqueName: \"kubernetes.io/projected/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-kube-api-access-w4g76\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.755605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.755573 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4g76\" (UniqueName: \"kubernetes.io/projected/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-kube-api-access-w4g76\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.755817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.755689 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.755817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.755718 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.756390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.756365 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.758500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.758476 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.771261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.771239 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4g76\" (UniqueName: \"kubernetes.io/projected/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-kube-api-access-w4g76\") pod \"maas-api-revoke-sub-7ea2b688-sjp97\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:36.902944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.902862 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:36.903400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.903339 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:36.904224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.904206 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:20:36.932150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:36.932113 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:37.227578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.227551 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97"] Nov 04 01:20:37.228334 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:37.228311 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac097ca4_142b_4dbc_8764_0ff9f2b5427b.slice/crio-69f9c339118b1acddbba525bd39e420554c88c6b18c741b0db661f39af0baf96 WatchSource:0}: Error finding container 69f9c339118b1acddbba525bd39e420554c88c6b18c741b0db661f39af0baf96: Status 404 returned error can't find the container with id 69f9c339118b1acddbba525bd39e420554c88c6b18c741b0db661f39af0baf96 Nov 04 01:20:37.301535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.301503 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" event={"ID":"ac097ca4-142b-4dbc-8764-0ff9f2b5427b","Type":"ContainerStarted","Data":"69f9c339118b1acddbba525bd39e420554c88c6b18c741b0db661f39af0baf96"} Nov 04 01:20:37.431087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.431061 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:37.467370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.467335 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cf6ade-85bd-4c25-a923-4c041c046941-kube-api-access-hfsv9\" (UniqueName: \"kubernetes.io/projected/e9cf6ade-85bd-4c25-a923-4c041c046941-kube-api-access-hfsv9\") pod \"e9cf6ade-85bd-4c25-a923-4c041c046941\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " Nov 04 01:20:37.467518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.467388 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-openshift-ca\") pod \"e9cf6ade-85bd-4c25-a923-4c041c046941\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " Nov 04 01:20:37.467518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.467508 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-xks-ca\") pod \"e9cf6ade-85bd-4c25-a923-4c041c046941\" (UID: \"e9cf6ade-85bd-4c25-a923-4c041c046941\") " Nov 04 01:20:37.468653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.468621 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-openshift-ca" pod "e9cf6ade-85bd-4c25-a923-4c041c046941" (UID: "e9cf6ade-85bd-4c25-a923-4c041c046941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:37.470390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.470362 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cf6ade-85bd-4c25-a923-4c041c046941-kube-api-access-hfsv9" pod "e9cf6ade-85bd-4c25-a923-4c041c046941" (UID: "e9cf6ade-85bd-4c25-a923-4c041c046941"). InnerVolumeSpecName "kube-api-access-hfsv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:37.470584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.470563 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-xks-ca" pod "e9cf6ade-85bd-4c25-a923-4c041c046941" (UID: "e9cf6ade-85bd-4c25-a923-4c041c046941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:37.568922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.568892 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:37.568922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.568924 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfsv9\" (UniqueName: \"kubernetes.io/projected/e9cf6ade-85bd-4c25-a923-4c041c046941-kube-api-access-hfsv9\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:37.569151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:37.568940 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cf6ade-85bd-4c25-a923-4c041c046941-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:38.309866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:38.309842 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" Nov 04 01:20:38.310250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:38.309882 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc" event={"ID":"e9cf6ade-85bd-4c25-a923-4c041c046941","Type":"ContainerDied","Data":"e24317b6c1b7313600d65e466fa6f039036288e6dd2b1f15842bb1eea0cc14bc"} Nov 04 01:20:38.310250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:38.309934 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e24317b6c1b7313600d65e466fa6f039036288e6dd2b1f15842bb1eea0cc14bc" Nov 04 01:20:38.311624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:38.311603 5051 generic.go:356] "Generic (PLEG): container finished" podID="ac097ca4-142b-4dbc-8764-0ff9f2b5427b" containerID="ac1083756ab320e96fd9d47840914ce4fdcd09312ccafe4b22163b5331500968" exitCode=0 Nov 04 01:20:38.311759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:38.311682 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" event={"ID":"ac097ca4-142b-4dbc-8764-0ff9f2b5427b","Type":"ContainerDied","Data":"ac1083756ab320e96fd9d47840914ce4fdcd09312ccafe4b22163b5331500968"} Nov 04 01:20:39.480714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.480683 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5"] Nov 04 01:20:39.481250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.481233 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e9cf6ade-85bd-4c25-a923-4c041c046941" containerName="revoke-keys" Nov 04 01:20:39.481250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.481249 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cf6ade-85bd-4c25-a923-4c041c046941" containerName="revoke-keys" Nov 04 01:20:39.481391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.481362 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cf6ade-85bd-4c25-a923-4c041c046941" containerName="revoke-keys" Nov 04 01:20:39.485208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.485189 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.488597 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.488568 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:39.493689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.493658 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5"] Nov 04 01:20:39.535377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.535337 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.535532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.535393 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwfz\" (UniqueName: \"kubernetes.io/projected/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-kube-api-access-ptwfz\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.535580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.535532 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.636173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.636072 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-xks-ca\") pod \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " Nov 04 01:20:39.636173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.636112 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-kube-api-access-w4g76\" (UniqueName: \"kubernetes.io/projected/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-kube-api-access-w4g76\") pod \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " Nov 04 01:20:39.636406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.636334 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-openshift-ca\") pod \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\" (UID: \"ac097ca4-142b-4dbc-8764-0ff9f2b5427b\") " Nov 04 01:20:39.636651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.636629 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.636740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.636661 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwfz\" (UniqueName: \"kubernetes.io/projected/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-kube-api-access-ptwfz\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.636740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.636687 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-openshift-ca" pod "ac097ca4-142b-4dbc-8764-0ff9f2b5427b" (UID: "ac097ca4-142b-4dbc-8764-0ff9f2b5427b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:39.636890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.636807 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.636962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.636916 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:39.637491 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.637465 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.638329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.638300 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-xks-ca" pod "ac097ca4-142b-4dbc-8764-0ff9f2b5427b" (UID: "ac097ca4-142b-4dbc-8764-0ff9f2b5427b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:39.638423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.638367 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-kube-api-access-w4g76" pod "ac097ca4-142b-4dbc-8764-0ff9f2b5427b" (UID: "ac097ca4-142b-4dbc-8764-0ff9f2b5427b"). InnerVolumeSpecName "kube-api-access-w4g76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:39.639013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.638996 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.644380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.644359 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwfz\" (UniqueName: \"kubernetes.io/projected/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-kube-api-access-ptwfz\") pod \"maas-api-revoke-sub-f31cbc7d-v28b5\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:39.737572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.737534 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:39.737572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.737576 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4g76\" (UniqueName: \"kubernetes.io/projected/ac097ca4-142b-4dbc-8764-0ff9f2b5427b-kube-api-access-w4g76\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:39.823211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:39.823184 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:40.109519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.109494 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5"] Nov 04 01:20:40.110112 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:40.110072 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a2fed7_5c0d_4b3b_98be_9928ccc7f0ab.slice/crio-7cb5d6ad276357fe4e68d7e493629753a3ed6cac0abef663bf67bca62bb24123 WatchSource:0}: Error finding container 7cb5d6ad276357fe4e68d7e493629753a3ed6cac0abef663bf67bca62bb24123: Status 404 returned error can't find the container with id 7cb5d6ad276357fe4e68d7e493629753a3ed6cac0abef663bf67bca62bb24123 Nov 04 01:20:40.198262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.198229 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk"] Nov 04 01:20:40.199029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.199005 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ac097ca4-142b-4dbc-8764-0ff9f2b5427b" containerName="revoke-keys" Nov 04 01:20:40.199029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.199028 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac097ca4-142b-4dbc-8764-0ff9f2b5427b" containerName="revoke-keys" Nov 04 01:20:40.199264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.199246 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac097ca4-142b-4dbc-8764-0ff9f2b5427b" containerName="revoke-keys" Nov 04 01:20:40.204589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.204566 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.212115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.210671 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"openshift-service-ca.crt\"" Nov 04 01:20:40.212115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.210822 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"kube-root-ca.crt\"" Nov 04 01:20:40.212115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.211046 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"distinct-w4-099dd9-kserve-self-signed-certs\"" Nov 04 01:20:40.212115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.211255 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-099dd9\"/\"default-dockercfg-jnlg9\"" Nov 04 01:20:40.213414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.213369 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk"] Nov 04 01:20:40.242641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.242616 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-home\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.242761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.242645 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-model-cache\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.242966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.242934 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tmp-dir\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.243104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.243083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tls-certs\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.243299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.243277 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-dshm\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.243375 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.243353 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2csf\" (UniqueName: \"kubernetes.io/projected/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-kube-api-access-t2csf\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.327308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.327149 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" event={"ID":"ac097ca4-142b-4dbc-8764-0ff9f2b5427b","Type":"ContainerDied","Data":"69f9c339118b1acddbba525bd39e420554c88c6b18c741b0db661f39af0baf96"} Nov 04 01:20:40.327308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.327212 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69f9c339118b1acddbba525bd39e420554c88c6b18c741b0db661f39af0baf96" Nov 04 01:20:40.327308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.327161 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97" Nov 04 01:20:40.328903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.328875 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" event={"ID":"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab","Type":"ContainerStarted","Data":"7cb5d6ad276357fe4e68d7e493629753a3ed6cac0abef663bf67bca62bb24123"} Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.344524 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-home\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.344572 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-model-cache\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.344729 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tmp-dir\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.344754 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tls-certs\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.344827 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-dshm\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.344873 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2csf\" (UniqueName: \"kubernetes.io/projected/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-kube-api-access-t2csf\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.345511 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-home\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.345780 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-model-cache\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.346504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.346012 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tmp-dir\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.375852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.375006 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-dshm\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.379826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.379759 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tls-certs\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.383680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.383606 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2csf\" (UniqueName: \"kubernetes.io/projected/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-kube-api-access-t2csf\") pod \"distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.521008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.520978 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:40.810554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.809807 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk"] Nov 04 01:20:40.819027 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:40.818982 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15e4c8bf_0ad3_436a_85cc_a9f31a94458b.slice/crio-15f1f1a1f84e3adc1bf34d427a55bd9cd236810e87d67b5839b582077061fee4 WatchSource:0}: Error finding container 15f1f1a1f84e3adc1bf34d427a55bd9cd236810e87d67b5839b582077061fee4: Status 404 returned error can't find the container with id 15f1f1a1f84e3adc1bf34d427a55bd9cd236810e87d67b5839b582077061fee4 Nov 04 01:20:40.828415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.827964 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl"] Nov 04 01:20:40.852188 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.852164 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:40.856362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.856300 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl"] Nov 04 01:20:40.951433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.951405 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brll4\" (UniqueName: \"kubernetes.io/projected/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-kube-api-access-brll4\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:40.951566 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.951484 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:40.951633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:40.951581 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:41.052551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.052513 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brll4\" (UniqueName: \"kubernetes.io/projected/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-kube-api-access-brll4\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:41.052706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.052615 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:41.052754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.052710 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:41.053297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.053272 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:41.055040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.055017 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:41.060232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.060207 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brll4\" (UniqueName: \"kubernetes.io/projected/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-kube-api-access-brll4\") pod \"maas-api-revoke-sub-acfe3388-fnvdl\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:41.204672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.204607 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:41.336829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.336794 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" event={"ID":"15e4c8bf-0ad3-436a-85cc-a9f31a94458b","Type":"ContainerStarted","Data":"54bcbdabcb6926bbe327c008340e7f0f384134c49a4db0af7a61296b3a0d804e"} Nov 04 01:20:41.337010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.336842 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" event={"ID":"15e4c8bf-0ad3-436a-85cc-a9f31a94458b","Type":"ContainerStarted","Data":"15f1f1a1f84e3adc1bf34d427a55bd9cd236810e87d67b5839b582077061fee4"} Nov 04 01:20:41.339713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.339675 5051 generic.go:356] "Generic (PLEG): container finished" podID="a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab" containerID="4856d251165a9c75bfed0f1533c2ecc0e446b95d7225759afb0fe8bed6fc8e3a" exitCode=0 Nov 04 01:20:41.339862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.339835 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" event={"ID":"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab","Type":"ContainerDied","Data":"4856d251165a9c75bfed0f1533c2ecc0e446b95d7225759afb0fe8bed6fc8e3a"} Nov 04 01:20:41.355616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.355572 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podStartSLOduration=1.355557723 podStartE2EDuration="1.355557723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:41.352403342 +0000 UTC m=+1380.324152213" watchObservedRunningTime="2026-11-04 01:20:41.355557723 +0000 UTC m=+1380.327306595" Nov 04 01:20:41.520097 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:41.520066 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b39b24_5d37_4443_b5aa_6efd51aa3b04.slice/crio-6e47a410acd29eac11b05deec71165012cbd11978fd06d604d0ba095cc682d32 WatchSource:0}: Error finding container 6e47a410acd29eac11b05deec71165012cbd11978fd06d604d0ba095cc682d32: Status 404 returned error can't find the container with id 6e47a410acd29eac11b05deec71165012cbd11978fd06d604d0ba095cc682d32 Nov 04 01:20:41.521362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.521340 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:41.521714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:41.521661 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl"] Nov 04 01:20:42.390564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.390515 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" event={"ID":"b1b39b24-5d37-4443-b5aa-6efd51aa3b04","Type":"ContainerStarted","Data":"b29fe3d36b2abfb261d0c02ded7bfd88e01cbebb2f370c86cdada5dc4342bf18"} Nov 04 01:20:42.390832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.390803 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" event={"ID":"b1b39b24-5d37-4443-b5aa-6efd51aa3b04","Type":"ContainerStarted","Data":"6e47a410acd29eac11b05deec71165012cbd11978fd06d604d0ba095cc682d32"} Nov 04 01:20:42.417233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.417176 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" podStartSLOduration=2.417159401 podStartE2EDuration="2.417159401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:42.413927936 +0000 UTC m=+1381.385676807" watchObservedRunningTime="2026-11-04 01:20:42.417159401 +0000 UTC m=+1381.388908275" Nov 04 01:20:42.563849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.563823 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:42.570575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.570536 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-kube-api-access-ptwfz\" (UniqueName: \"kubernetes.io/projected/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-kube-api-access-ptwfz\") pod \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " Nov 04 01:20:42.570719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.570703 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-openshift-ca\") pod \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " Nov 04 01:20:42.570873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.570845 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-xks-ca\") pod \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\" (UID: \"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab\") " Nov 04 01:20:42.571596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.571562 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-openshift-ca" pod "a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab" (UID: "a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:42.573667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.573644 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-xks-ca" pod "a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab" (UID: "a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:42.573853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.573830 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-kube-api-access-ptwfz" pod "a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab" (UID: "a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab"). InnerVolumeSpecName "kube-api-access-ptwfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:42.672046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.671953 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:42.672046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.671986 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwfz\" (UniqueName: \"kubernetes.io/projected/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-kube-api-access-ptwfz\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:42.672046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:42.672003 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:43.401622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:43.401587 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" Nov 04 01:20:43.401622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:43.401613 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5" event={"ID":"a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab","Type":"ContainerDied","Data":"7cb5d6ad276357fe4e68d7e493629753a3ed6cac0abef663bf67bca62bb24123"} Nov 04 01:20:43.401872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:43.401674 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cb5d6ad276357fe4e68d7e493629753a3ed6cac0abef663bf67bca62bb24123" Nov 04 01:20:43.403610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:43.403579 5051 generic.go:356] "Generic (PLEG): container finished" podID="b1b39b24-5d37-4443-b5aa-6efd51aa3b04" containerID="b29fe3d36b2abfb261d0c02ded7bfd88e01cbebb2f370c86cdada5dc4342bf18" exitCode=0 Nov 04 01:20:43.403794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:43.403766 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" event={"ID":"b1b39b24-5d37-4443-b5aa-6efd51aa3b04","Type":"ContainerDied","Data":"b29fe3d36b2abfb261d0c02ded7bfd88e01cbebb2f370c86cdada5dc4342bf18"} Nov 04 01:20:44.891152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:44.879503 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:45.000817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:44.986939 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-kube-api-access-brll4\" (UniqueName: \"kubernetes.io/projected/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-kube-api-access-brll4\") pod \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " Nov 04 01:20:45.000817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:44.987082 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-openshift-ca\") pod \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " Nov 04 01:20:45.000817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:44.987225 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-xks-ca\") pod \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\" (UID: \"b1b39b24-5d37-4443-b5aa-6efd51aa3b04\") " Nov 04 01:20:45.031145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.024176 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-openshift-ca" pod "b1b39b24-5d37-4443-b5aa-6efd51aa3b04" (UID: "b1b39b24-5d37-4443-b5aa-6efd51aa3b04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:45.045296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.044827 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-xks-ca" pod "b1b39b24-5d37-4443-b5aa-6efd51aa3b04" (UID: "b1b39b24-5d37-4443-b5aa-6efd51aa3b04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:45.045296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.045231 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-kube-api-access-brll4" pod "b1b39b24-5d37-4443-b5aa-6efd51aa3b04" (UID: "b1b39b24-5d37-4443-b5aa-6efd51aa3b04"). InnerVolumeSpecName "kube-api-access-brll4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:45.097429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.096996 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:45.097429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.097037 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:45.097429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.097052 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brll4\" (UniqueName: \"kubernetes.io/projected/b1b39b24-5d37-4443-b5aa-6efd51aa3b04-kube-api-access-brll4\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:45.523710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.522709 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" event={"ID":"b1b39b24-5d37-4443-b5aa-6efd51aa3b04","Type":"ContainerDied","Data":"6e47a410acd29eac11b05deec71165012cbd11978fd06d604d0ba095cc682d32"} Nov 04 01:20:45.523710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.522773 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e47a410acd29eac11b05deec71165012cbd11978fd06d604d0ba095cc682d32" Nov 04 01:20:45.523710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:45.522843 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl" Nov 04 01:20:47.076135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.076074 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9"] Nov 04 01:20:47.077022 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.076997 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b1b39b24-5d37-4443-b5aa-6efd51aa3b04" containerName="revoke-keys" Nov 04 01:20:47.077186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.077025 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b39b24-5d37-4443-b5aa-6efd51aa3b04" containerName="revoke-keys" Nov 04 01:20:47.077186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.077069 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab" containerName="revoke-keys" Nov 04 01:20:47.077186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.077077 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab" containerName="revoke-keys" Nov 04 01:20:47.077354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.077263 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b39b24-5d37-4443-b5aa-6efd51aa3b04" containerName="revoke-keys" Nov 04 01:20:47.077354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.077285 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab" containerName="revoke-keys" Nov 04 01:20:47.081550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.081530 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.090980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.090797 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9"] Nov 04 01:20:47.091148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.091097 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:20:47.091224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.091114 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:47.091319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.091302 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:20:47.091365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.091321 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:20:47.168233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.168200 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.168424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.168240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnfm4\" (UniqueName: \"kubernetes.io/projected/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-kube-api-access-wnfm4\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.168424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.168262 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.276016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.270801 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.276016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.270855 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnfm4\" (UniqueName: \"kubernetes.io/projected/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-kube-api-access-wnfm4\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.276016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.270886 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.289222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.278297 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.289222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.281313 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.294714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.294649 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnfm4\" (UniqueName: \"kubernetes.io/projected/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-kube-api-access-wnfm4\") pod \"maas-api-revoke-sub-ee1c1373-9d8q9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.398727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.398652 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:47.679584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:47.679546 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9"] Nov 04 01:20:47.680374 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:47.680344 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2f6b72_cb78_41d0_ac5a_e5d302a713b9.slice/crio-5ceb399f22f0eb01a579833b32338ca798fed0684798905d364798c24590fb4f WatchSource:0}: Error finding container 5ceb399f22f0eb01a579833b32338ca798fed0684798905d364798c24590fb4f: Status 404 returned error can't find the container with id 5ceb399f22f0eb01a579833b32338ca798fed0684798905d364798c24590fb4f Nov 04 01:20:48.596878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:48.596846 5051 generic.go:356] "Generic (PLEG): container finished" podID="4a2f6b72-cb78-41d0-ac5a-e5d302a713b9" containerID="75fd2fd5b58c24b90e4d9c6a0a54e517f0f7e841dbdfab89a2046e5003acb736" exitCode=0 Nov 04 01:20:48.597275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:48.597019 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" event={"ID":"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9","Type":"ContainerDied","Data":"75fd2fd5b58c24b90e4d9c6a0a54e517f0f7e841dbdfab89a2046e5003acb736"} Nov 04 01:20:48.597275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:48.597056 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" event={"ID":"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9","Type":"ContainerStarted","Data":"5ceb399f22f0eb01a579833b32338ca798fed0684798905d364798c24590fb4f"} Nov 04 01:20:49.680951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.680912 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4"] Nov 04 01:20:49.685204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.685179 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.690023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.689996 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4"] Nov 04 01:20:49.703550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.703214 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.703550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.703269 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6dn\" (UniqueName: \"kubernetes.io/projected/48f8ed22-7c81-47b6-a0fc-d52b366f305f-kube-api-access-hm6dn\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.703550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.703414 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.804209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.804160 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.804355 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.804219 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6dn\" (UniqueName: \"kubernetes.io/projected/48f8ed22-7c81-47b6-a0fc-d52b366f305f-kube-api-access-hm6dn\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.804433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.804391 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.804954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.804925 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.807335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.807313 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.813367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.813344 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6dn\" (UniqueName: \"kubernetes.io/projected/48f8ed22-7c81-47b6-a0fc-d52b366f305f-kube-api-access-hm6dn\") pod \"maas-api-revoke-sub-8daa67e6-grqt4\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:49.833164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.833140 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:49.905293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.905198 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-xks-ca\") pod \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " Nov 04 01:20:49.905293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.905238 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-kube-api-access-wnfm4\" (UniqueName: \"kubernetes.io/projected/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-kube-api-access-wnfm4\") pod \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " Nov 04 01:20:49.905525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.905301 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-openshift-ca\") pod \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\" (UID: \"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9\") " Nov 04 01:20:49.905795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.905763 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-openshift-ca" pod "4a2f6b72-cb78-41d0-ac5a-e5d302a713b9" (UID: "4a2f6b72-cb78-41d0-ac5a-e5d302a713b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:49.905983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.905963 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:49.907829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.907802 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-kube-api-access-wnfm4" pod "4a2f6b72-cb78-41d0-ac5a-e5d302a713b9" (UID: "4a2f6b72-cb78-41d0-ac5a-e5d302a713b9"). InnerVolumeSpecName "kube-api-access-wnfm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:49.908156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:49.908085 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-xks-ca" pod "4a2f6b72-cb78-41d0-ac5a-e5d302a713b9" (UID: "4a2f6b72-cb78-41d0-ac5a-e5d302a713b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:50.007876 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.007826 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:50.007876 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.007863 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnfm4\" (UniqueName: \"kubernetes.io/projected/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9-kube-api-access-wnfm4\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:50.008224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.007994 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:50.011233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.011045 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k"] Nov 04 01:20:50.011898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.011871 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4a2f6b72-cb78-41d0-ac5a-e5d302a713b9" containerName="revoke-keys" Nov 04 01:20:50.011898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.011899 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2f6b72-cb78-41d0-ac5a-e5d302a713b9" containerName="revoke-keys" Nov 04 01:20:50.012135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.012107 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2f6b72-cb78-41d0-ac5a-e5d302a713b9" containerName="revoke-keys" Nov 04 01:20:50.034216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.034180 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.038795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.038762 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k"] Nov 04 01:20:50.039445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.039421 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"openshift-service-ca.crt\"" Nov 04 01:20:50.039874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.039840 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"kube-root-ca.crt\"" Nov 04 01:20:50.040438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.040409 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"default-dockercfg-zwvp4\"" Nov 04 01:20:50.041327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.041307 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"distinct-w3-ca55c1-kserve-self-signed-certs\"" Nov 04 01:20:50.109114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.109087 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tls-certs\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.109241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.109211 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpjc\" (UniqueName: \"kubernetes.io/projected/06e23a20-9d9c-4335-899b-03f9e8eee4b9-kube-api-access-kcpjc\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.109306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.109277 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-model-cache\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.109404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.109384 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tmp-dir\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.109464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.109429 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-home\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.109499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.109487 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-dshm\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.211194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.211060 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tls-certs\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.211194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.211110 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpjc\" (UniqueName: \"kubernetes.io/projected/06e23a20-9d9c-4335-899b-03f9e8eee4b9-kube-api-access-kcpjc\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.211194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.211190 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-model-cache\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.211467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.211237 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tmp-dir\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.211467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.211268 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-home\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.211467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.211288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-dshm\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.212217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.212191 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-model-cache\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.212520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.212489 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tmp-dir\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.212805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.212785 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-home\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.216846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.216812 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tls-certs\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.218507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.217654 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-dshm\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.242015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.241975 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpjc\" (UniqueName: \"kubernetes.io/projected/06e23a20-9d9c-4335-899b-03f9e8eee4b9-kube-api-access-kcpjc\") pod \"distinct-w3-ca55c1-kserve-59ccb57444-gjx6k\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.323162 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:50.323097 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f8ed22_7c81_47b6_a0fc_d52b366f305f.slice/crio-fe60b9197ddc9f13e255d8ef1c6019fa49041a8b50942436995a78ca9bcc9a0a WatchSource:0}: Error finding container fe60b9197ddc9f13e255d8ef1c6019fa49041a8b50942436995a78ca9bcc9a0a: Status 404 returned error can't find the container with id fe60b9197ddc9f13e255d8ef1c6019fa49041a8b50942436995a78ca9bcc9a0a Nov 04 01:20:50.323162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.323133 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4"] Nov 04 01:20:50.355835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.355804 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:50.522215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.522171 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:50.536833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.536797 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:50.612487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.612457 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" Nov 04 01:20:50.612654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.612490 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9" event={"ID":"4a2f6b72-cb78-41d0-ac5a-e5d302a713b9","Type":"ContainerDied","Data":"5ceb399f22f0eb01a579833b32338ca798fed0684798905d364798c24590fb4f"} Nov 04 01:20:50.612654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.612540 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ceb399f22f0eb01a579833b32338ca798fed0684798905d364798c24590fb4f" Nov 04 01:20:50.614187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.614156 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" event={"ID":"48f8ed22-7c81-47b6-a0fc-d52b366f305f","Type":"ContainerStarted","Data":"fe60b9197ddc9f13e255d8ef1c6019fa49041a8b50942436995a78ca9bcc9a0a"} Nov 04 01:20:50.627563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.627538 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:20:50.639740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:50.639716 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k"] Nov 04 01:20:50.641280 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:50.641255 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06e23a20_9d9c_4335_899b_03f9e8eee4b9.slice/crio-27dab2c839f8615e4b24f95f522a6d078206af36666560ac8c34d71a874294c2 WatchSource:0}: Error finding container 27dab2c839f8615e4b24f95f522a6d078206af36666560ac8c34d71a874294c2: Status 404 returned error can't find the container with id 27dab2c839f8615e4b24f95f522a6d078206af36666560ac8c34d71a874294c2 Nov 04 01:20:51.625249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:51.625110 5051 generic.go:356] "Generic (PLEG): container finished" podID="48f8ed22-7c81-47b6-a0fc-d52b366f305f" containerID="38b2f08d5df526774f1672d8ee92674323c61716ab8c7c4444fe6200924244a6" exitCode=0 Nov 04 01:20:51.625249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:51.625203 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" event={"ID":"48f8ed22-7c81-47b6-a0fc-d52b366f305f","Type":"ContainerDied","Data":"38b2f08d5df526774f1672d8ee92674323c61716ab8c7c4444fe6200924244a6"} Nov 04 01:20:51.627213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:51.627186 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" event={"ID":"06e23a20-9d9c-4335-899b-03f9e8eee4b9","Type":"ContainerStarted","Data":"1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588"} Nov 04 01:20:51.627329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:51.627224 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" event={"ID":"06e23a20-9d9c-4335-899b-03f9e8eee4b9","Type":"ContainerStarted","Data":"27dab2c839f8615e4b24f95f522a6d078206af36666560ac8c34d71a874294c2"} Nov 04 01:20:51.658542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:51.658493 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podStartSLOduration=2.6584780500000003 podStartE2EDuration="2.65847805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:51.657842621 +0000 UTC m=+1390.629591492" watchObservedRunningTime="2026-11-04 01:20:51.65847805 +0000 UTC m=+1390.630226920" Nov 04 01:20:52.356114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.356077 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:20:52.642178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.642152 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj"] Nov 04 01:20:52.646321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.646300 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.660572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.655198 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj"] Nov 04 01:20:52.755196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.754655 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.755196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.754819 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.755196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.754947 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtklr\" (UniqueName: \"kubernetes.io/projected/52d8a0be-a45e-421e-816e-9e11997d83e1-kube-api-access-qtklr\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.856760 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.856729 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.856905 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.856879 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.857032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.857017 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtklr\" (UniqueName: \"kubernetes.io/projected/52d8a0be-a45e-421e-816e-9e11997d83e1-kube-api-access-qtklr\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.857711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.857681 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.859754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.859723 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.861941 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.861916 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:52.871273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.871252 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtklr\" (UniqueName: \"kubernetes.io/projected/52d8a0be-a45e-421e-816e-9e11997d83e1-kube-api-access-qtklr\") pod \"maas-api-revoke-sub-c1cdb602-hn9bj\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:52.958368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.958273 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-xks-ca\") pod \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " Nov 04 01:20:52.958527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.958394 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-openshift-ca\") pod \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " Nov 04 01:20:52.958527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.958431 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f8ed22-7c81-47b6-a0fc-d52b366f305f-kube-api-access-hm6dn\" (UniqueName: \"kubernetes.io/projected/48f8ed22-7c81-47b6-a0fc-d52b366f305f-kube-api-access-hm6dn\") pod \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\" (UID: \"48f8ed22-7c81-47b6-a0fc-d52b366f305f\") " Nov 04 01:20:52.958843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.958808 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-openshift-ca" pod "48f8ed22-7c81-47b6-a0fc-d52b366f305f" (UID: "48f8ed22-7c81-47b6-a0fc-d52b366f305f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:52.959004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.958991 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:52.960577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.960553 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-xks-ca" pod "48f8ed22-7c81-47b6-a0fc-d52b366f305f" (UID: "48f8ed22-7c81-47b6-a0fc-d52b366f305f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:52.960719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.960700 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f8ed22-7c81-47b6-a0fc-d52b366f305f-kube-api-access-hm6dn" pod "48f8ed22-7c81-47b6-a0fc-d52b366f305f" (UID: "48f8ed22-7c81-47b6-a0fc-d52b366f305f"). InnerVolumeSpecName "kube-api-access-hm6dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:52.981021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:52.980998 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:53.060908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.060873 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f8ed22-7c81-47b6-a0fc-d52b366f305f-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:53.060908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.060913 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6dn\" (UniqueName: \"kubernetes.io/projected/48f8ed22-7c81-47b6-a0fc-d52b366f305f-kube-api-access-hm6dn\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:53.263195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.263156 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj"] Nov 04 01:20:53.263580 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:53.263553 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d8a0be_a45e_421e_816e_9e11997d83e1.slice/crio-364e76fac87598513ad9d5fbc3ce7df41e62d5830ed3fd50917469253ef8dab1 WatchSource:0}: Error finding container 364e76fac87598513ad9d5fbc3ce7df41e62d5830ed3fd50917469253ef8dab1: Status 404 returned error can't find the container with id 364e76fac87598513ad9d5fbc3ce7df41e62d5830ed3fd50917469253ef8dab1 Nov 04 01:20:53.650978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.650936 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" Nov 04 01:20:53.651441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.651002 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4" event={"ID":"48f8ed22-7c81-47b6-a0fc-d52b366f305f","Type":"ContainerDied","Data":"fe60b9197ddc9f13e255d8ef1c6019fa49041a8b50942436995a78ca9bcc9a0a"} Nov 04 01:20:53.651441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.651073 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe60b9197ddc9f13e255d8ef1c6019fa49041a8b50942436995a78ca9bcc9a0a" Nov 04 01:20:53.653113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.653084 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" event={"ID":"52d8a0be-a45e-421e-816e-9e11997d83e1","Type":"ContainerStarted","Data":"1fa0c94f72e69f2edcbc3b2e162d94421abf366c368dfbef77bb5b54505df398"} Nov 04 01:20:53.653113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.653156 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" event={"ID":"52d8a0be-a45e-421e-816e-9e11997d83e1","Type":"ContainerStarted","Data":"364e76fac87598513ad9d5fbc3ce7df41e62d5830ed3fd50917469253ef8dab1"} Nov 04 01:20:53.675895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:53.675854 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" podStartSLOduration=1.6758401410000001 podStartE2EDuration="1.675840141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:53.670926109 +0000 UTC m=+1392.642674981" watchObservedRunningTime="2026-11-04 01:20:53.675840141 +0000 UTC m=+1392.647589011" Nov 04 01:20:54.662268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:54.662235 5051 generic.go:356] "Generic (PLEG): container finished" podID="52d8a0be-a45e-421e-816e-9e11997d83e1" containerID="1fa0c94f72e69f2edcbc3b2e162d94421abf366c368dfbef77bb5b54505df398" exitCode=0 Nov 04 01:20:54.662696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:54.662299 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" event={"ID":"52d8a0be-a45e-421e-816e-9e11997d83e1","Type":"ContainerDied","Data":"1fa0c94f72e69f2edcbc3b2e162d94421abf366c368dfbef77bb5b54505df398"} Nov 04 01:20:55.772817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.772781 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8"] Nov 04 01:20:55.773605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.773453 5051 cpu_manager.go:414] "containerMap: removing container" podUID="48f8ed22-7c81-47b6-a0fc-d52b366f305f" containerName="revoke-keys" Nov 04 01:20:55.773605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.773471 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f8ed22-7c81-47b6-a0fc-d52b366f305f" containerName="revoke-keys" Nov 04 01:20:55.773605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.773578 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f8ed22-7c81-47b6-a0fc-d52b366f305f" containerName="revoke-keys" Nov 04 01:20:55.779070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.779044 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:55.789187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.786811 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"openshift-service-ca.crt\"" Nov 04 01:20:55.789187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.787992 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"distinct-2-w1-586541-kserve-self-signed-certs\"" Nov 04 01:20:55.789187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.788179 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"default-dockercfg-26p4v\"" Nov 04 01:20:55.800150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.793655 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-586541\"/\"kube-root-ca.crt\"" Nov 04 01:20:55.800828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.800803 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8"] Nov 04 01:20:55.871651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.871625 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:55.907175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.906376 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d8a0be-a45e-421e-816e-9e11997d83e1-kube-api-access-qtklr\" (UniqueName: \"kubernetes.io/projected/52d8a0be-a45e-421e-816e-9e11997d83e1-kube-api-access-qtklr\") pod \"52d8a0be-a45e-421e-816e-9e11997d83e1\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " Nov 04 01:20:55.907175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.906449 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-openshift-ca\") pod \"52d8a0be-a45e-421e-816e-9e11997d83e1\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " Nov 04 01:20:55.907175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.906479 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-xks-ca\") pod \"52d8a0be-a45e-421e-816e-9e11997d83e1\" (UID: \"52d8a0be-a45e-421e-816e-9e11997d83e1\") " Nov 04 01:20:55.907175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.906753 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1af8632-7d07-4b0a-b1f6-872816217eed-tls-certs\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:55.907175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.906815 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-tmp-dir\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:55.907175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.906909 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-home\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:55.908072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.907618 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-model-cache\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:55.908072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.907754 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzxz\" (UniqueName: \"kubernetes.io/projected/f1af8632-7d07-4b0a-b1f6-872816217eed-kube-api-access-rxzxz\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:55.908072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.907812 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-dshm\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:55.923420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.913702 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-xks-ca" pod "52d8a0be-a45e-421e-816e-9e11997d83e1" (UID: "52d8a0be-a45e-421e-816e-9e11997d83e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:20:55.923833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.923764 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d8a0be-a45e-421e-816e-9e11997d83e1-kube-api-access-qtklr" pod "52d8a0be-a45e-421e-816e-9e11997d83e1" (UID: "52d8a0be-a45e-421e-816e-9e11997d83e1"). InnerVolumeSpecName "kube-api-access-qtklr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:20:55.943192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:55.941325 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-openshift-ca" pod "52d8a0be-a45e-421e-816e-9e11997d83e1" (UID: "52d8a0be-a45e-421e-816e-9e11997d83e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:20:56.009982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.009942 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1af8632-7d07-4b0a-b1f6-872816217eed-tls-certs\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.010169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010029 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-tmp-dir\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.010169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010116 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-home\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.010337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010199 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-model-cache\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.010337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010321 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzxz\" (UniqueName: \"kubernetes.io/projected/f1af8632-7d07-4b0a-b1f6-872816217eed-kube-api-access-rxzxz\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.010443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010366 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-dshm\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.010492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010446 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:56.010492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010462 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d8a0be-a45e-421e-816e-9e11997d83e1-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:56.010612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010478 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtklr\" (UniqueName: \"kubernetes.io/projected/52d8a0be-a45e-421e-816e-9e11997d83e1-kube-api-access-qtklr\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:20:56.010891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.010865 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-home\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.011108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.011088 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-tmp-dir\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.011338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.011319 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-model-cache\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.013229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.013210 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1af8632-7d07-4b0a-b1f6-872816217eed-tls-certs\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.013304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.013228 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-dshm\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.020621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.020595 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzxz\" (UniqueName: \"kubernetes.io/projected/f1af8632-7d07-4b0a-b1f6-872816217eed-kube-api-access-rxzxz\") pod \"distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.140560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.140529 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:56.427906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.427879 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8"] Nov 04 01:20:56.428567 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:56.428542 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1af8632_7d07_4b0a_b1f6_872816217eed.slice/crio-3fbfdc423a2f4e15f48ea89dbc0d5c474ffeccc8a277a176b6fbd96cf4e78f20 WatchSource:0}: Error finding container 3fbfdc423a2f4e15f48ea89dbc0d5c474ffeccc8a277a176b6fbd96cf4e78f20: Status 404 returned error can't find the container with id 3fbfdc423a2f4e15f48ea89dbc0d5c474ffeccc8a277a176b6fbd96cf4e78f20 Nov 04 01:20:56.679667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.679625 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" event={"ID":"f1af8632-7d07-4b0a-b1f6-872816217eed","Type":"ContainerStarted","Data":"5fda7a886dbc3cf20238cfd4b0fc315b6b03ddfd46f3ee9f560ba828cce1d26f"} Nov 04 01:20:56.679830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.679678 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" event={"ID":"f1af8632-7d07-4b0a-b1f6-872816217eed","Type":"ContainerStarted","Data":"3fbfdc423a2f4e15f48ea89dbc0d5c474ffeccc8a277a176b6fbd96cf4e78f20"} Nov 04 01:20:56.681241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.681211 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" Nov 04 01:20:56.681365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.681254 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj" event={"ID":"52d8a0be-a45e-421e-816e-9e11997d83e1","Type":"ContainerDied","Data":"364e76fac87598513ad9d5fbc3ce7df41e62d5830ed3fd50917469253ef8dab1"} Nov 04 01:20:56.681365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.681298 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="364e76fac87598513ad9d5fbc3ce7df41e62d5830ed3fd50917469253ef8dab1" Nov 04 01:20:56.702690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:56.702602 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podStartSLOduration=1.702589782 podStartE2EDuration="1.702589782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:56.700754762 +0000 UTC m=+1395.672503637" watchObservedRunningTime="2026-11-04 01:20:56.702589782 +0000 UTC m=+1395.674338710" Nov 04 01:20:57.141070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.141032 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:20:57.268166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.268130 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp"] Nov 04 01:20:57.268727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.268710 5051 cpu_manager.go:414] "containerMap: removing container" podUID="52d8a0be-a45e-421e-816e-9e11997d83e1" containerName="revoke-keys" Nov 04 01:20:57.268727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.268727 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d8a0be-a45e-421e-816e-9e11997d83e1" containerName="revoke-keys" Nov 04 01:20:57.268850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.268816 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d8a0be-a45e-421e-816e-9e11997d83e1" containerName="revoke-keys" Nov 04 01:20:57.273056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.273036 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.295372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.293046 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp"] Nov 04 01:20:57.344537 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344498 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.344737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344605 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.344737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344662 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.344737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344688 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.344737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344720 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.344977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344827 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlth\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-kube-api-access-kdlth\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.344977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344857 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.344977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344900 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.344977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.344974 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.387700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.387664 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt"] Nov 04 01:20:57.392735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.392670 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.404021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.403963 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt"] Nov 04 01:20:57.446323 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446297 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446336 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446408 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446450 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446501 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446541 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446565 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446608 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446714 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlth\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-kube-api-access-kdlth\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446768 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446878 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.446993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.446981 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.447153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.447059 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.447495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.447474 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.448733 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.448706 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.449053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.449037 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.454726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.454699 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.455095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.455072 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlth\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-kube-api-access-kdlth\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.499725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.499675 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s"] Nov 04 01:20:57.506007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.505985 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.517704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.517679 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s"] Nov 04 01:20:57.548211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548174 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.548400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548236 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.548400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548253 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20b9fc8-c661-4782-9739-33c09ef7b15c-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.548400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548339 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.548400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548354 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsh5c\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-kube-api-access-wsh5c\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.548400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548378 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.548667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548408 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.548667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548437 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.548667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.548456 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.591140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.591086 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:57.623114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.623079 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9"] Nov 04 01:20:57.629397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.629367 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.644233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.643563 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9"] Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.654313 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl"] Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655505 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655553 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655580 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20b9fc8-c661-4782-9739-33c09ef7b15c-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655604 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655659 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655781 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655807 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsh5c\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-kube-api-access-wsh5c\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655840 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655883 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wx5\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-kube-api-access-l8wx5\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655928 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.655953 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.656001 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.656029 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.656089 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.656167 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.656198 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.656270 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.656327 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.657926 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20b9fc8-c661-4782-9739-33c09ef7b15c-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.658654 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.659289 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.659494 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.659565 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.660163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.659824 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.665636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.664648 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.678054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.677591 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.678054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.677712 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.680746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.680705 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl"] Nov 04 01:20:57.710266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.710218 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsh5c\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-kube-api-access-wsh5c\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.713003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.712811 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:57.757820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.757769 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.757998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.757828 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.757998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.757883 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.757998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.757909 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.757998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.757938 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.757998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.757964 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.757998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.757998 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.758352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758041 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.758352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758070 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.758352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758189 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.758352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758222 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.758352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758251 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.758352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758281 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.758352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758304 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.758681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758366 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.758681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758409 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.758681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758436 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.758681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758474 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.758681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758586 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.758681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758623 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.758681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758655 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjddn\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-kube-api-access-jjddn\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.759027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758683 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmfh\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-kube-api-access-7vmfh\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.759027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758724 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wx5\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-kube-api-access-l8wx5\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.759027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758750 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.759027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758786 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.759027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758828 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.759027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.758897 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.759459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.759435 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.760532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.760507 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.761272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.761249 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.763468 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.763440 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.763793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.763762 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.764576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.764527 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.765601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.765494 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.772023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.771979 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wx5\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-kube-api-access-l8wx5\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.795401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.789605 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.828367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.828336 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:57.861049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.861013 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.861295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.861270 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.862087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.862065 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjddn\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-kube-api-access-jjddn\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.862248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.862229 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmfh\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-kube-api-access-7vmfh\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.862376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.862359 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.862537 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.862477 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.862731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.862716 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.862838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.861749 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.863071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.863048 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.864005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.863982 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.864093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.863333 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.864173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.863917 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.864219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.864157 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.864219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.864195 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.865180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.865158 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.865225 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.864498 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.865264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.865083 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.865363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.864724 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.865631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.865613 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.866371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.866334 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.866694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.866680 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.867324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.867309 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.867800 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.867071 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.867941 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.866910 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.868105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.867691 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.868207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.868188 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.868278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.868223 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.868278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.868262 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.869157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.869098 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.869577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.869349 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.869901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.869880 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.872606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.872535 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjddn\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-kube-api-access-jjddn\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.872980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.872935 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.873921 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.873875 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmfh\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-kube-api-access-7vmfh\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.874078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.874058 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:57.884473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.884194 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:57.916345 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.916308 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp"] Nov 04 01:20:57.916935 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:57.916902 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a8e42a_646e_46c6_ad74_0b54acadfeae.slice/crio-c225eb15a86edfe5322d4c6a7290501cdbdee56271d031b5c3a953c8d4cfe3aa WatchSource:0}: Error finding container c225eb15a86edfe5322d4c6a7290501cdbdee56271d031b5c3a953c8d4cfe3aa: Status 404 returned error can't find the container with id c225eb15a86edfe5322d4c6a7290501cdbdee56271d031b5c3a953c8d4cfe3aa Nov 04 01:20:57.919669 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.919635 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:57.919719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.919704 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:57.919750 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.919731 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:57.979573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:57.979385 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:58.001387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.001205 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:58.026837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.026606 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt"] Nov 04 01:20:58.027692 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:58.027665 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20b9fc8_c661_4782_9739_33c09ef7b15c.slice/crio-cb3757e805a3087f51d7b3b200774e7801f1a3d6cb31e0cc51640ca1c775aff8 WatchSource:0}: Error finding container cb3757e805a3087f51d7b3b200774e7801f1a3d6cb31e0cc51640ca1c775aff8: Status 404 returned error can't find the container with id cb3757e805a3087f51d7b3b200774e7801f1a3d6cb31e0cc51640ca1c775aff8 Nov 04 01:20:58.031303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.030790 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.031303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.030864 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.031303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.030911 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.163267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.163217 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s"] Nov 04 01:20:58.173638 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:20:58.173590 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70adf13e_81e4_46a0_96ae_09ba8b3ba474.slice/crio-1e174c81a9a22bd684f5da5b78cee600cf25023838b14bd7fd84737873e1423f WatchSource:0}: Error finding container 1e174c81a9a22bd684f5da5b78cee600cf25023838b14bd7fd84737873e1423f: Status 404 returned error can't find the container with id 1e174c81a9a22bd684f5da5b78cee600cf25023838b14bd7fd84737873e1423f Nov 04 01:20:58.181424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.177722 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.181424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.177838 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.181424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.177888 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.339654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.338647 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9"] Nov 04 01:20:58.349296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.349256 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.349434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.349349 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.349434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.349396 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.353833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.353523 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl"] Nov 04 01:20:58.358158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.357861 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.358158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.357938 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.358158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.358004 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:20:58.738787 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.738750 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" event={"ID":"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7","Type":"ContainerStarted","Data":"77901f77b583c6c2ee9014488fcc8d20f3f1c4f25981c5e3b6a0bcf73b0ade8c"} Nov 04 01:20:58.740134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.739019 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" event={"ID":"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7","Type":"ContainerStarted","Data":"ca69a642cf3e3e5bfe4fa8e1d528dd2feec780fb0a837b5fda0d16691a68793a"} Nov 04 01:20:58.742353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.742296 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" event={"ID":"b20b9fc8-c661-4782-9739-33c09ef7b15c","Type":"ContainerStarted","Data":"83438193f00a9a199f20d489f8c4e3fb7aef72898b4aef9bf302433a96d17874"} Nov 04 01:20:58.742353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.742347 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" event={"ID":"b20b9fc8-c661-4782-9739-33c09ef7b15c","Type":"ContainerStarted","Data":"cb3757e805a3087f51d7b3b200774e7801f1a3d6cb31e0cc51640ca1c775aff8"} Nov 04 01:20:58.744617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.744588 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" event={"ID":"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30","Type":"ContainerStarted","Data":"7da8a305e495bb78e08f160c2152fa1142422fa8be22b6f95719f9f17cc8e3cf"} Nov 04 01:20:58.744813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.744791 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" event={"ID":"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30","Type":"ContainerStarted","Data":"feb798862e0ef08a4f74cfdbad4cd3f4b124439ec1f460bf03be844965fab268"} Nov 04 01:20:58.747537 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.747512 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" event={"ID":"e0a8e42a-646e-46c6-ad74-0b54acadfeae","Type":"ContainerStarted","Data":"828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc"} Nov 04 01:20:58.747704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.747687 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" event={"ID":"e0a8e42a-646e-46c6-ad74-0b54acadfeae","Type":"ContainerStarted","Data":"c225eb15a86edfe5322d4c6a7290501cdbdee56271d031b5c3a953c8d4cfe3aa"} Nov 04 01:20:58.751742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.751695 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" event={"ID":"70adf13e-81e4-46a0-96ae-09ba8b3ba474","Type":"ContainerStarted","Data":"1ff057d8e1cf9711c11cc3195c497f4879a0b532e87737751897c5b0e6ef2a11"} Nov 04 01:20:58.751913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.751894 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" event={"ID":"70adf13e-81e4-46a0-96ae-09ba8b3ba474","Type":"ContainerStarted","Data":"1e174c81a9a22bd684f5da5b78cee600cf25023838b14bd7fd84737873e1423f"} Nov 04 01:20:58.792923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.792857 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podStartSLOduration=1.792837032 podStartE2EDuration="1.792837032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.770717176 +0000 UTC m=+1397.742466048" watchObservedRunningTime="2026-11-04 01:20:58.792837032 +0000 UTC m=+1397.764585904" Nov 04 01:20:58.829910 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.829871 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:20:58.833642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.833584 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podStartSLOduration=1.833564306 podStartE2EDuration="1.833564306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.82006557 +0000 UTC m=+1397.791814445" watchObservedRunningTime="2026-11-04 01:20:58.833564306 +0000 UTC m=+1397.805313176" Nov 04 01:20:58.848230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.848170 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podStartSLOduration=1.848152351 podStartE2EDuration="1.848152351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.84225199 +0000 UTC m=+1397.814000862" watchObservedRunningTime="2026-11-04 01:20:58.848152351 +0000 UTC m=+1397.819901223" Nov 04 01:20:58.905146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.884425 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podStartSLOduration=1.8844099600000002 podStartE2EDuration="1.88440996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.883280261 +0000 UTC m=+1397.855029134" watchObservedRunningTime="2026-11-04 01:20:58.88440996 +0000 UTC m=+1397.856158832" Nov 04 01:20:58.929305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.929204 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podStartSLOduration=1.929185087 podStartE2EDuration="1.929185087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:20:58.924403426 +0000 UTC m=+1397.896152298" watchObservedRunningTime="2026-11-04 01:20:58.929185087 +0000 UTC m=+1397.900933960" Nov 04 01:20:58.982148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:58.981309 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:20:59.011931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.011847 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:20:59.012374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.012341 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.012576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.012551 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:20:59.324177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.324076 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5"] Nov 04 01:20:59.331328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.331298 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.331583 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.331561 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5"] Nov 04 01:20:59.363145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.339657 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:20:59.363145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.339900 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:20:59.363145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.340024 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:20:59.363145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.340912 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:20:59.465282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.465240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srh9s\" (UniqueName: \"kubernetes.io/projected/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-kube-api-access-srh9s\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.476154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.465686 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.496145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.480415 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.627151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.616290 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srh9s\" (UniqueName: \"kubernetes.io/projected/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-kube-api-access-srh9s\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.627151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.616399 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.627151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.616550 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.627151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.617561 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.676148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.653229 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:20:59.703281 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.703243 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.703530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.703496 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:20:59.771227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.736194 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:20:59.815204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.810686 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srh9s\" (UniqueName: \"kubernetes.io/projected/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-kube-api-access-srh9s\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.838144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.832143 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.838144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.832202 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:20:59.838144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.832857 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab1ea1a8-p6rq5\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:20:59.929216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.910005 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:20:59.961190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.955503 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:00.000151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.997232 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.000151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:20:59.997300 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:00.086167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.079031 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.086167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.079141 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:00.186204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.170707 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:21:00.395620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.395585 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:21:00.445193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.444230 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:00.491852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.491807 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:00.665145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.646735 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.665145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.646802 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:00.805980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.731758 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.805980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.731832 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:00.846528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.828874 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:00.846528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:00.828931 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:01.015272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.015213 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.015588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.015538 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:01.094235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.057745 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.094235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.057805 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:01.465197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.395959 5051 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:01.527151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.526253 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:01.527151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.527020 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:01.551138 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.548067 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:21:01.570762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.560228 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:01.596611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.592739 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.596611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.592796 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:01.657272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.548113 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5"] Nov 04 01:21:01.657272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.611309 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:21:01.657272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.617826 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:01.718371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.718285 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.718603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.718574 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:01.975475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.974869 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:01.975475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:01.974932 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:02.066619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.066583 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.066896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.066860 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:02.067729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.067687 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.067924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.067901 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:02.481053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.481018 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" event={"ID":"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc","Type":"ContainerStarted","Data":"26e17eb1fc273cc9ae8367c30343c88e460425f951f185609f8c2b27ee53085f"} Nov 04 01:21:02.539990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.539952 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.560653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.560612 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.592602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.592568 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.592855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.592829 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:02.618310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.618267 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:02.619872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.619816 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:21:02.726244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.723338 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.726244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.723415 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:02.915147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.848256 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:02.915147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:02.848322 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:03.016147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.012741 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.016147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.012806 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:03.016147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.013618 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.016147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.013656 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:03.564616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.550228 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:03.564616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.563049 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:03.609725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.609261 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.609725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.609327 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:03.634600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.634221 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:03.702159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.689335 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" event={"ID":"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc","Type":"ContainerStarted","Data":"7dfc10ccf4dbd7f65c4ca2dfd3f1662086b3bf9e4e3fce5dcfb960b5253b40b7"} Nov 04 01:21:03.729177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.720032 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" podStartSLOduration=4.72001225 podStartE2EDuration="4.72001225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:03.717332853 +0000 UTC m=+1402.689081726" watchObservedRunningTime="2026-11-04 01:21:03.72001225 +0000 UTC m=+1402.691761124" Nov 04 01:21:03.733144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.729797 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.742236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.742194 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:03.916211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.909266 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.916211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.909337 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:03.984150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.983472 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:03.984150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:03.983531 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:04.004153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.003789 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.004153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.003843 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:04.352483 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:21:04.352444 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:21:04.578162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.572211 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.578162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.573810 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.596938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.596902 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.597368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.597335 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:04.680217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.680105 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:04.749432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.717893 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.749432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.717951 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:04.861150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.857035 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:04.861150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.857092 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:04.995478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:04.995441 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.031673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.031632 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:05.032319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.031552 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.032497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.032474 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:05.033816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.033793 5051 generic.go:356] "Generic (PLEG): container finished" podID="f54fc5de-a9de-4a8a-88a8-473dff5f9cfc" containerID="7dfc10ccf4dbd7f65c4ca2dfd3f1662086b3bf9e4e3fce5dcfb960b5253b40b7" exitCode=0 Nov 04 01:21:05.033958 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.033940 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" event={"ID":"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc","Type":"ContainerDied","Data":"7dfc10ccf4dbd7f65c4ca2dfd3f1662086b3bf9e4e3fce5dcfb960b5253b40b7"} Nov 04 01:21:05.603147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.578094 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:21:05.628157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.626237 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.628157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.626297 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:05.643157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.642262 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.701146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.696736 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:05.728803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.728765 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.729049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.729021 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:05.837146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.836473 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:21:05.837146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.837137 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:05.837412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:05.837185 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:06.008560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.008522 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.008852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.008812 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:06.013144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.009638 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.017142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.013315 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:06.187146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.147211 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:21:06.771001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.770961 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:06.773680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.771888 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:06.806149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.797389 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.806149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.797450 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:06.806149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.798679 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.806149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.798719 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:06.823275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.822242 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:06.837602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.837481 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:06.837602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:06.837534 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:07.017721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.012597 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.017721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.012664 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:07.017721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.012784 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.017721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.012810 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:07.238732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.167701 5051 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.780056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.780011 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.786153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.786105 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.786351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.786326 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:07.786863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.786834 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:07.871521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.871475 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:07.899998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.894162 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.899998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.894222 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:07.899998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.894300 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.899998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.894321 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:07.994482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.994450 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:07.994847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:07.994816 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:08.058566 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.058475 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.060799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.060767 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:08.171146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.170393 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:21:08.308341 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.306742 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-xks-ca\") pod \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " Nov 04 01:21:08.308341 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.306836 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-openshift-ca\") pod \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " Nov 04 01:21:08.308341 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.306996 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-kube-api-access-srh9s\" (UniqueName: \"kubernetes.io/projected/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-kube-api-access-srh9s\") pod \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\" (UID: \"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc\") " Nov 04 01:21:08.321031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.319490 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:21:08.333376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.333339 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-openshift-ca" pod "f54fc5de-a9de-4a8a-88a8-473dff5f9cfc" (UID: "f54fc5de-a9de-4a8a-88a8-473dff5f9cfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:08.351163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.350326 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-kube-api-access-srh9s" pod "f54fc5de-a9de-4a8a-88a8-473dff5f9cfc" (UID: "f54fc5de-a9de-4a8a-88a8-473dff5f9cfc"). InnerVolumeSpecName "kube-api-access-srh9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:08.401703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.401637 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-xks-ca" pod "f54fc5de-a9de-4a8a-88a8-473dff5f9cfc" (UID: "f54fc5de-a9de-4a8a-88a8-473dff5f9cfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:08.465169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.409199 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:08.465169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.409236 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:08.465169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.409250 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srh9s\" (UniqueName: \"kubernetes.io/projected/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc-kube-api-access-srh9s\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:08.600703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.600608 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.600988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.600939 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:08.789592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.789546 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:08.810545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.797338 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:08.827404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.821793 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.827404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.821843 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:08.844154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.838724 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" event={"ID":"f54fc5de-a9de-4a8a-88a8-473dff5f9cfc","Type":"ContainerDied","Data":"26e17eb1fc273cc9ae8367c30343c88e460425f951f185609f8c2b27ee53085f"} Nov 04 01:21:08.844154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.838781 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26e17eb1fc273cc9ae8367c30343c88e460425f951f185609f8c2b27ee53085f" Nov 04 01:21:08.844154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.838847 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5" Nov 04 01:21:08.912451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.912417 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.912675 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.912649 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:08.993248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.993214 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:08.993512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:08.993481 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:09.043344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.043296 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.043639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.043608 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:09.394677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.388886 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 04 01:21:09.595157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.594293 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.595157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.594372 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:09.608060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.607686 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:21:09.791145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.790934 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:21:09.862974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.862937 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.863250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.863220 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:09.877005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.876962 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:09.924174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.901790 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:09.924174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:09.901851 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:10.283211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.022731 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.368447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.368405 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:10.408145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.055360 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.408145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.388591 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:10.601096 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.601005 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.601371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.601341 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:10.617146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.613687 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:10.637331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.637205 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:21:10.722568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.722527 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.722832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.722801 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:10.946611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.946511 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:10.959252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.947145 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:10.994217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.971478 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.994217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.971533 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:10.994217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.988841 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:10.994217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:10.988900 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:11.019753 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.019716 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:11.020034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.020001 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:11.403145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.369832 5051 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:11.564144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.559249 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:21:11.632494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.632446 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:21:11.632741 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.632708 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 04 01:21:11.632946 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.632919 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 04 01:21:11.724208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.717013 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:11.724208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.717082 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:11.830935 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.830885 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:11.840279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.840235 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:11.987201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.985006 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:11.987201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.985067 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:11.987856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.987824 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:21:11.995148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.988244 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:12.000148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:11.995700 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:21:12.066146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.020667 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:12.113693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.113654 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:12.381198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.377361 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx"] Nov 04 01:21:12.381198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.378347 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f54fc5de-a9de-4a8a-88a8-473dff5f9cfc" containerName="revoke-keys" Nov 04 01:21:12.381198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.378371 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54fc5de-a9de-4a8a-88a8-473dff5f9cfc" containerName="revoke-keys" Nov 04 01:21:12.381198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.378561 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54fc5de-a9de-4a8a-88a8-473dff5f9cfc" containerName="revoke-keys" Nov 04 01:21:12.475142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.474423 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.560591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.560555 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx"] Nov 04 01:21:12.707552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.707466 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:21:12.757049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.731314 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:21:12.757049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.756432 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:12.757049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.756497 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:12.759055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.759010 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:12.784847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.784815 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.785344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.785086 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.785509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.785493 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.788791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.786508 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.809566 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.809534 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.809927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.809818 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.810481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.810457 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.833342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.833211 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plntc\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-kube-api-access-plntc\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.833906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.833881 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.835833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.835723 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:21:12.845552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.845517 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:12.845774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.845746 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:12.942084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.941429 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.944146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.943589 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.946997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.946533 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.948900 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.948859 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.949957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.949934 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.962192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.950240 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.993143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.973199 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plntc\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-kube-api-access-plntc\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.993143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.973334 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.993143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.973519 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.993143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.962598 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.993143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.953038 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.993143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.975392 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.993143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.976479 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:12.993143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.976734 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:13.016145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.993999 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:13.016145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:12.994205 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:21:13.016145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.000208 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:13.016145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.000270 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:13.016145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.000827 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:13.016145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.009213 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:21:13.041145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.021656 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:13.067145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.042018 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plntc\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-kube-api-access-plntc\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:13.067145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.047435 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:13.070853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.070484 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 04 01:21:13.070853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.070543 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 04 01:21:13.132875 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.126963 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:13.271720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.269912 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:21:13.271720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.269979 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:21:13.282052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.282019 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:21:13.517296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.517257 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:13.718151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.714730 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Nov 04 01:21:13.718151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.714788 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Nov 04 01:21:13.747870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.747258 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:13.747870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.747327 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:13.884199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.830665 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 04 01:21:13.884199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.830724 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 04 01:21:13.985942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.985578 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 01:21:13.985942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:13.985638 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 01:21:14.797336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:14.797192 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx"] Nov 04 01:21:14.798734 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:21:14.798705 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6697cf69_03fa_480f_b8d7_263e7d5d84c7.slice/crio-2dda14088fd520706aaa5bf248fe5dbf8c1e69a7e13ee77445ec308ab004e9cf WatchSource:0}: Error finding container 2dda14088fd520706aaa5bf248fe5dbf8c1e69a7e13ee77445ec308ab004e9cf: Status 404 returned error can't find the container with id 2dda14088fd520706aaa5bf248fe5dbf8c1e69a7e13ee77445ec308ab004e9cf Nov 04 01:21:14.801915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:14.801872 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:14.802488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:14.802461 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:14.802666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:14.802647 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:14.810382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:14.810316 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:14.810382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:14.810375 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:15.020252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.020214 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:15.020558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.020524 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:15.280960 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.261999 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g"] Nov 04 01:21:15.365404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.365366 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g"] Nov 04 01:21:15.371141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.369106 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.392978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.392892 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:21:15.393337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.393314 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:21:15.419146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.406326 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:21:15.419146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.406427 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:21:15.443224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.441137 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.443224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.441222 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.443224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.441438 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntv5p\" (UniqueName: \"kubernetes.io/projected/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-kube-api-access-ntv5p\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.464143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.461937 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" event={"ID":"6697cf69-03fa-480f-b8d7-263e7d5d84c7","Type":"ContainerStarted","Data":"2dda14088fd520706aaa5bf248fe5dbf8c1e69a7e13ee77445ec308ab004e9cf"} Nov 04 01:21:15.549336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.549297 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.549632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.549607 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.550055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.550031 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntv5p\" (UniqueName: \"kubernetes.io/projected/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-kube-api-access-ntv5p\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.551847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.551820 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.554956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.554905 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.561726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.561203 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntv5p\" (UniqueName: \"kubernetes.io/projected/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-kube-api-access-ntv5p\") pod \"maas-api-revoke-sub-2f477b14-vb52g\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.691411 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.677179 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:15.747158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.741269 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:15.747158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.741344 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:15.853675 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.852919 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:15.853675 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.852984 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:15.856146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.855635 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:21:15.857111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.856340 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:15.857111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.856389 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:15.858829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.858770 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:21:15.886415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.882101 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:21:15.919146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:15.919048 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:21:16.124426 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.124384 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:16.124642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.124457 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:16.468152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.458449 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:21:16.711206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.704078 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" event={"ID":"6697cf69-03fa-480f-b8d7-263e7d5d84c7","Type":"ContainerStarted","Data":"8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d"} Nov 04 01:21:16.841289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.788621 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:16.841289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.788685 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:16.870196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.870166 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:21:16.976219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.944265 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:21:16.987696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:16.987661 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:17.013289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.013244 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:17.152358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.148590 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:21:17.184580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.184546 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g"] Nov 04 01:21:17.186935 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.186878 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podStartSLOduration=5.18686024 podStartE2EDuration="5.18686024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:17.165980427 +0000 UTC m=+1416.137729300" watchObservedRunningTime="2026-11-04 01:21:17.18686024 +0000 UTC m=+1416.158609113" Nov 04 01:21:17.363535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.348602 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:21:17.363535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.359084 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:21:17.417663 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.417297 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:21:17.446338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.446314 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:21:17.518902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.518815 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:17.526517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.526473 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:21:17.526921 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.526882 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:21:17.726355 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.726320 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" event={"ID":"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb","Type":"ContainerStarted","Data":"828035fc1eebbdf0d0d5e4d0c1845f4120c049cdf8fd0bf40620727ebca65291"} Nov 04 01:21:17.726636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.726612 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" event={"ID":"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb","Type":"ContainerStarted","Data":"589c801bdd19a69d67da6b4c5deb7c7b93514af363dd4467495d0a9f0f5e34ae"} Nov 04 01:21:17.781319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:17.781212 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" podStartSLOduration=2.7811937909999997 podStartE2EDuration="2.781193791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:17.749312789 +0000 UTC m=+1416.721061662" watchObservedRunningTime="2026-11-04 01:21:17.781193791 +0000 UTC m=+1416.752942661" Nov 04 01:21:18.518432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:18.518397 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:21:18.518868 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:18.518456 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:21:18.739061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:18.739026 5051 generic.go:356] "Generic (PLEG): container finished" podID="83864e06-2d7d-40c0-8d8c-48d8c58d3cdb" containerID="828035fc1eebbdf0d0d5e4d0c1845f4120c049cdf8fd0bf40620727ebca65291" exitCode=0 Nov 04 01:21:18.739382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:18.739149 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" event={"ID":"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb","Type":"ContainerDied","Data":"828035fc1eebbdf0d0d5e4d0c1845f4120c049cdf8fd0bf40620727ebca65291"} Nov 04 01:21:19.289474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.289075 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz"] Nov 04 01:21:19.294723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.294696 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.304812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.304779 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz"] Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.317072 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"kube-root-ca.crt\"" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.317164 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"default-dockercfg-dhtjx\"" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.318004 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"openshift-service-ca.crt\"" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.319572 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee876a\"/\"distinct-w0-ee876a-kserve-self-signed-certs\"" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.320973 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45230ed9-1e0d-4579-a9da-4be008858d09-tls-certs\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.321034 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-tmp-dir\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.321205 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnnn\" (UniqueName: \"kubernetes.io/projected/45230ed9-1e0d-4579-a9da-4be008858d09-kube-api-access-zbnnn\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.321269 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-home\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.321373 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-dshm\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.322166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.321417 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-model-cache\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.428144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.423368 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-home\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.428144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.423528 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-dshm\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.428144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.423555 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-model-cache\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.428144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.423614 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45230ed9-1e0d-4579-a9da-4be008858d09-tls-certs\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.428144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.423677 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-tmp-dir\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.428144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.423810 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnnn\" (UniqueName: \"kubernetes.io/projected/45230ed9-1e0d-4579-a9da-4be008858d09-kube-api-access-zbnnn\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.430325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.429750 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-model-cache\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.430325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.430022 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-home\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.453687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.438943 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-tmp-dir\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.453687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.447886 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45230ed9-1e0d-4579-a9da-4be008858d09-tls-certs\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.517967 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.517929 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnnn\" (UniqueName: \"kubernetes.io/projected/45230ed9-1e0d-4579-a9da-4be008858d09-kube-api-access-zbnnn\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.522142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.518702 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-dshm\") pod \"distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:19.522763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.522737 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:21:19.522920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.522900 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:21:19.621466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:19.621374 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:20.138391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.138307 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz"] Nov 04 01:21:20.148861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.148839 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:20.318018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.317986 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-kube-api-access-ntv5p\" (UniqueName: \"kubernetes.io/projected/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-kube-api-access-ntv5p\") pod \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " Nov 04 01:21:20.318213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.318192 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-xks-ca\") pod \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " Nov 04 01:21:20.318376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.318357 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-openshift-ca\") pod \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\" (UID: \"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb\") " Nov 04 01:21:20.319434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.319393 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-openshift-ca" pod "83864e06-2d7d-40c0-8d8c-48d8c58d3cdb" (UID: "83864e06-2d7d-40c0-8d8c-48d8c58d3cdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:20.320680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.320651 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-kube-api-access-ntv5p" pod "83864e06-2d7d-40c0-8d8c-48d8c58d3cdb" (UID: "83864e06-2d7d-40c0-8d8c-48d8c58d3cdb"). InnerVolumeSpecName "kube-api-access-ntv5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:20.321636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.321619 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-xks-ca" pod "83864e06-2d7d-40c0-8d8c-48d8c58d3cdb" (UID: "83864e06-2d7d-40c0-8d8c-48d8c58d3cdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:20.399223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.399193 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:21:20.411218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.411187 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:21:20.420038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.420007 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:20.420038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.420036 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:20.420234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.420052 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntv5p\" (UniqueName: \"kubernetes.io/projected/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb-kube-api-access-ntv5p\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:20.519565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.518576 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:21:20.519565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.518683 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:21:20.789283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.789165 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" event={"ID":"83864e06-2d7d-40c0-8d8c-48d8c58d3cdb","Type":"ContainerDied","Data":"589c801bdd19a69d67da6b4c5deb7c7b93514af363dd4467495d0a9f0f5e34ae"} Nov 04 01:21:20.789283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.789269 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="589c801bdd19a69d67da6b4c5deb7c7b93514af363dd4467495d0a9f0f5e34ae" Nov 04 01:21:20.789821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.789352 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g" Nov 04 01:21:20.792365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.792338 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" event={"ID":"45230ed9-1e0d-4579-a9da-4be008858d09","Type":"ContainerStarted","Data":"2b7d340067d428ddfae56ceb1c465eddb6333d3d278cd6b4d696ba409c9c5375"} Nov 04 01:21:20.792507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.792373 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" event={"ID":"45230ed9-1e0d-4579-a9da-4be008858d09","Type":"ContainerStarted","Data":"b7c24d48e59cda9865cbb1bca850ec0392390f44a350f61c7d0ee4535c423612"} Nov 04 01:21:20.816243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:20.816195 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podStartSLOduration=1.816181758 podStartE2EDuration="1.816181758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:20.812326008 +0000 UTC m=+1419.784074877" watchObservedRunningTime="2026-11-04 01:21:20.816181758 +0000 UTC m=+1419.787930622" Nov 04 01:21:21.518830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:21.518797 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:21:21.518979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:21.518852 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:21:21.622084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:21.622046 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:22.521622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:22.521582 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 04 01:21:22.522505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:22.522451 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 04 01:21:23.522826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:23.522792 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:23.523284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:23.523082 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:23.523820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:23.523801 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:21:28.416144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:28.405203 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.475144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:28.473225 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:21:28.566325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:28.566261 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:28.641310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:28.641254 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.224140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.224078 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.381278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.381223 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.408300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.408248 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.409302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.409268 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:21:29.495291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.473459 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.495291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.474305 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:21:29.691298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.691244 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:29.716174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.712096 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:21:29.770238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.743428 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:21:29.969148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.851469 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:29.969148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:29.908213 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:21:30.529678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:30.494233 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:30.609226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:30.607601 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:30.609226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:30.607690 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:30.685835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:30.685778 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:21:30.771379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:30.740620 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:31.355166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.049187 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.355675 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.049363 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:21:31.356063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.154609 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:31.356280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.208526 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:31.382791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.356243 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.444630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.444571 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.572461 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.563166 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:21:31.704210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.495205 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.704210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.642036 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:31.704210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.670654 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:31.953146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.953078 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:21:31.953146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.953145 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:31.953414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:31.953179 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:21:32.376142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.263551 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:32.376142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.263628 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:32.376142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.364436 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:21:32.376142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.364509 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:32.436264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.430512 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:32.453084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.430675 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:32.855245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.855213 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:21:32.914272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.914223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:32.923810 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.923781 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:21:32.954111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:32.954063 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:21:33.021705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.021670 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:21:33.321867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.321829 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m"] Nov 04 01:21:33.322473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.322452 5051 cpu_manager.go:414] "containerMap: removing container" podUID="83864e06-2d7d-40c0-8d8c-48d8c58d3cdb" containerName="revoke-keys" Nov 04 01:21:33.322473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.322471 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83864e06-2d7d-40c0-8d8c-48d8c58d3cdb" containerName="revoke-keys" Nov 04 01:21:33.322670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.322577 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="83864e06-2d7d-40c0-8d8c-48d8c58d3cdb" containerName="revoke-keys" Nov 04 01:21:33.326254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.326230 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.334629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.331918 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:21:33.334629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.331979 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:21:33.357319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.334779 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:21:33.357319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.334890 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:21:33.372142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.370549 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m"] Nov 04 01:21:33.417354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.417309 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.417511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.417386 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.417656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.417641 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snsqv\" (UniqueName: \"kubernetes.io/projected/4c1988c6-4378-46cf-be31-172ee5be96d5-kube-api-access-snsqv\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.521237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.521178 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snsqv\" (UniqueName: \"kubernetes.io/projected/4c1988c6-4378-46cf-be31-172ee5be96d5-kube-api-access-snsqv\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.521416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.521312 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.521416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.521368 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.525605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.525567 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.525605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.525597 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.534369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.534338 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snsqv\" (UniqueName: \"kubernetes.io/projected/4c1988c6-4378-46cf-be31-172ee5be96d5-kube-api-access-snsqv\") pod \"maas-api-revoke-sub-86667efb-cxk5m\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:33.676214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:33.671544 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:34.006334 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:21:34.006298 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1988c6_4378_46cf_be31_172ee5be96d5.slice/crio-a0c57bc1a807a13634e83eadd3e168189b4e3750d263dffb8e4fe39c0de53d8d WatchSource:0}: Error finding container a0c57bc1a807a13634e83eadd3e168189b4e3750d263dffb8e4fe39c0de53d8d: Status 404 returned error can't find the container with id a0c57bc1a807a13634e83eadd3e168189b4e3750d263dffb8e4fe39c0de53d8d Nov 04 01:21:34.006770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:34.006532 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m"] Nov 04 01:21:34.121522 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:21:34.121470 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:21:34.567895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:34.565619 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" event={"ID":"4c1988c6-4378-46cf-be31-172ee5be96d5","Type":"ContainerStarted","Data":"4d7e242da2bf3a9e4a1ba2902feda5690f596d3786bcb345ca4910d587a033f1"} Nov 04 01:21:34.567895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:34.565681 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" event={"ID":"4c1988c6-4378-46cf-be31-172ee5be96d5","Type":"ContainerStarted","Data":"a0c57bc1a807a13634e83eadd3e168189b4e3750d263dffb8e4fe39c0de53d8d"} Nov 04 01:21:34.588745 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:34.588668 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" podStartSLOduration=1.588650644 podStartE2EDuration="1.588650644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:34.587005884 +0000 UTC m=+1433.558754756" watchObservedRunningTime="2026-11-04 01:21:34.588650644 +0000 UTC m=+1433.560399516" Nov 04 01:21:35.576015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:35.575976 5051 generic.go:356] "Generic (PLEG): container finished" podID="4c1988c6-4378-46cf-be31-172ee5be96d5" containerID="4d7e242da2bf3a9e4a1ba2902feda5690f596d3786bcb345ca4910d587a033f1" exitCode=0 Nov 04 01:21:35.576502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:35.576061 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" event={"ID":"4c1988c6-4378-46cf-be31-172ee5be96d5","Type":"ContainerDied","Data":"4d7e242da2bf3a9e4a1ba2902feda5690f596d3786bcb345ca4910d587a033f1"} Nov 04 01:21:36.900046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:36.900016 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:36.974165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:36.974128 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-openshift-ca\") pod \"4c1988c6-4378-46cf-be31-172ee5be96d5\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " Nov 04 01:21:36.974364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:36.974227 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1988c6-4378-46cf-be31-172ee5be96d5-kube-api-access-snsqv\" (UniqueName: \"kubernetes.io/projected/4c1988c6-4378-46cf-be31-172ee5be96d5-kube-api-access-snsqv\") pod \"4c1988c6-4378-46cf-be31-172ee5be96d5\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " Nov 04 01:21:36.974449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:36.974424 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-xks-ca\") pod \"4c1988c6-4378-46cf-be31-172ee5be96d5\" (UID: \"4c1988c6-4378-46cf-be31-172ee5be96d5\") " Nov 04 01:21:36.974962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:36.974934 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-openshift-ca" pod "4c1988c6-4378-46cf-be31-172ee5be96d5" (UID: "4c1988c6-4378-46cf-be31-172ee5be96d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:21:36.977029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:36.977003 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1988c6-4378-46cf-be31-172ee5be96d5-kube-api-access-snsqv" pod "4c1988c6-4378-46cf-be31-172ee5be96d5" (UID: "4c1988c6-4378-46cf-be31-172ee5be96d5"). InnerVolumeSpecName "kube-api-access-snsqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:21:36.977116 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:36.977004 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-xks-ca" pod "4c1988c6-4378-46cf-be31-172ee5be96d5" (UID: "4c1988c6-4378-46cf-be31-172ee5be96d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:21:37.076777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:37.076739 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:37.076777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:37.076779 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snsqv\" (UniqueName: \"kubernetes.io/projected/4c1988c6-4378-46cf-be31-172ee5be96d5-kube-api-access-snsqv\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:37.076977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:37.076795 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c1988c6-4378-46cf-be31-172ee5be96d5-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:21:37.593917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:37.593884 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" Nov 04 01:21:37.593917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:37.593884 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m" event={"ID":"4c1988c6-4378-46cf-be31-172ee5be96d5","Type":"ContainerDied","Data":"a0c57bc1a807a13634e83eadd3e168189b4e3750d263dffb8e4fe39c0de53d8d"} Nov 04 01:21:37.594153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:37.593952 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0c57bc1a807a13634e83eadd3e168189b4e3750d263dffb8e4fe39c0de53d8d" Nov 04 01:21:42.502579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.502542 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s"] Nov 04 01:21:42.503171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.503155 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4c1988c6-4378-46cf-be31-172ee5be96d5" containerName="revoke-keys" Nov 04 01:21:42.503171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.503172 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1988c6-4378-46cf-be31-172ee5be96d5" containerName="revoke-keys" Nov 04 01:21:42.503306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.503297 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1988c6-4378-46cf-be31-172ee5be96d5" containerName="revoke-keys" Nov 04 01:21:42.508334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.508313 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.527667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.527629 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s"] Nov 04 01:21:42.589788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.589758 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.589991 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.589895 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c50b2-a884-4533-a240-cc18c3a65bb9-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.589991 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.589936 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpmk\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-kube-api-access-7kpmk\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.590133 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.590064 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.590196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.590146 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.590196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.590185 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.590306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.590216 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.590306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.590249 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.590306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.590273 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692071 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c50b2-a884-4533-a240-cc18c3a65bb9-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692138 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpmk\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-kube-api-access-7kpmk\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692201 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692260 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692295 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692354 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692386 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692409 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.692743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.692622 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.693385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.693344 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.693654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.693383 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.694243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.694222 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.694838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.694814 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c50b2-a884-4533-a240-cc18c3a65bb9-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.695440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.695419 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.698784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.698763 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.702535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.702489 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.704442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.704417 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw"] Nov 04 01:21:42.704532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.704482 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpmk\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-kube-api-access-7kpmk\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.716784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.716759 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.718564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.718536 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw"] Nov 04 01:21:42.720835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.720778 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.794574 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.794488 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.794734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.794608 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6zz\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-kube-api-access-8d6zz\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.794734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.794638 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.794734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.794680 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a3871f-4717-4043-807a-9200b309fe63-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.794851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.794785 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.794851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.794805 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.794964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.794941 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.795079 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.795054 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a3871f-4717-4043-807a-9200b309fe63-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.795157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.795113 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.838815 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.838779 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:42.896574 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896528 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6zz\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-kube-api-access-8d6zz\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.896574 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896578 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.896834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896623 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a3871f-4717-4043-807a-9200b309fe63-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.896834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896649 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.896834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896668 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.896834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896713 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.896834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896759 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a3871f-4717-4043-807a-9200b309fe63-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.896834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896784 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.897161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.896838 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.897389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.897364 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-certs\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.897579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.897554 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-data\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.897579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.897569 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.898046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.898019 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a3871f-4717-4043-807a-9200b309fe63-istiod-ca-cert\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.898311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.898289 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-credential-socket\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.902421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.902388 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-envoy\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.902747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.902719 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a3871f-4717-4043-807a-9200b309fe63-istio-podinfo\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.906849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.906814 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6zz\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-kube-api-access-8d6zz\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:42.907406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:42.907361 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-istio-token\") pod \"e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:43.058799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.058706 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:43.224004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.221223 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s"] Nov 04 01:21:43.232285 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:21:43.232227 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79c50b2_a884_4533_a240_cc18c3a65bb9.slice/crio-2d9b2450896a7db2553121345a1e5f0f958035e6734d252678b055c094beaa12 WatchSource:0}: Error finding container 2d9b2450896a7db2553121345a1e5f0f958035e6734d252678b055c094beaa12: Status 404 returned error can't find the container with id 2d9b2450896a7db2553121345a1e5f0f958035e6734d252678b055c094beaa12 Nov 04 01:21:43.235315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.235277 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:43.235429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.235347 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:43.235429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.235385 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:43.393435 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:21:43.393382 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a3871f_4717_4043_807a_9200b309fe63.slice/crio-1a549360af3763d84b642637ebfa2e9a80437a46d5ae6485fbb3a95849d997b4 WatchSource:0}: Error finding container 1a549360af3763d84b642637ebfa2e9a80437a46d5ae6485fbb3a95849d997b4: Status 404 returned error can't find the container with id 1a549360af3763d84b642637ebfa2e9a80437a46d5ae6485fbb3a95849d997b4 Nov 04 01:21:43.395216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.395190 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw"] Nov 04 01:21:43.397297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.397259 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:43.397484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.397456 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:43.397638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.397606 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:43.645500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.644645 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" event={"ID":"56a3871f-4717-4043-807a-9200b309fe63","Type":"ContainerStarted","Data":"0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4"} Nov 04 01:21:43.645500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.644724 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" event={"ID":"56a3871f-4717-4043-807a-9200b309fe63","Type":"ContainerStarted","Data":"1a549360af3763d84b642637ebfa2e9a80437a46d5ae6485fbb3a95849d997b4"} Nov 04 01:21:43.646807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.646782 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" event={"ID":"d79c50b2-a884-4533-a240-cc18c3a65bb9","Type":"ContainerStarted","Data":"e9656b4705c35f93e16133532d0ec840f29d296d2fd4732da334aa749c2e4210"} Nov 04 01:21:43.646996 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.646974 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" event={"ID":"d79c50b2-a884-4533-a240-cc18c3a65bb9","Type":"ContainerStarted","Data":"2d9b2450896a7db2553121345a1e5f0f958035e6734d252678b055c094beaa12"} Nov 04 01:21:43.672711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.672654 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podStartSLOduration=1.6726377129999999 podStartE2EDuration="1.672637713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:43.667094531 +0000 UTC m=+1442.638843402" watchObservedRunningTime="2026-11-04 01:21:43.672637713 +0000 UTC m=+1442.644386584" Nov 04 01:21:43.690872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.690640 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podStartSLOduration=1.690619598 podStartE2EDuration="1.690619598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:43.689844016 +0000 UTC m=+1442.661592887" watchObservedRunningTime="2026-11-04 01:21:43.690619598 +0000 UTC m=+1442.662368472" Nov 04 01:21:43.838984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:43.838949 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:44.068416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:44.068384 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:44.070284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:44.070190 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:44.070451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:44.070285 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:44.840515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:44.840482 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:21:44.840918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:44.840539 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 04 01:21:45.061549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:45.061510 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:45.062034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:45.062008 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:45.839713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:45.839679 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:21:45.839901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:45.839735 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 04 01:21:46.075146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:46.059664 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:46.075146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:46.059723 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:46.839139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:46.839096 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:21:46.839327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:46.839183 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 04 01:21:47.082146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.074245 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:47.082146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.074322 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:47.629888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.629850 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd"] Nov 04 01:21:47.638754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.638724 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:47.639445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.639421 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd"] Nov 04 01:21:47.643926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.643894 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ca55c1\"/\"distinct-2-w3-ca55c1-kserve-self-signed-certs\"" Nov 04 01:21:47.722687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.722447 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-dshm\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:47.723418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.723388 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-home\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:47.723746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.723727 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrx2\" (UniqueName: \"kubernetes.io/projected/01c41ff4-0264-4807-99e7-daa17e5d0f5a-kube-api-access-fgrx2\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:47.723962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.723942 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tls-certs\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:47.724134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.724102 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tmp-dir\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:47.724296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.724271 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-model-cache\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:47.883201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.856595 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:21:47.883570 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.883532 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:47.906749 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-dshm\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.010510 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-home\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.010967 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-home\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.011175 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrx2\" (UniqueName: \"kubernetes.io/projected/01c41ff4-0264-4807-99e7-daa17e5d0f5a-kube-api-access-fgrx2\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.011957 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tls-certs\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.012218 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tmp-dir\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.012286 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-model-cache\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.012475 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tmp-dir\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.027148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.012695 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-model-cache\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.044205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.033832 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tls-certs\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.044205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.038738 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-dshm\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.044205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.042313 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrx2\" (UniqueName: \"kubernetes.io/projected/01c41ff4-0264-4807-99e7-daa17e5d0f5a-kube-api-access-fgrx2\") pod \"distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.103144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.089327 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:48.112143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.109304 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:48.277576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.276259 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:48.821829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.821707 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd"] Nov 04 01:21:48.840202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.840108 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:21:48.840202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.840180 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 04 01:21:48.922666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:48.922636 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" event={"ID":"01c41ff4-0264-4807-99e7-daa17e5d0f5a","Type":"ContainerStarted","Data":"87caa55f2c796888e0d7e63f4afba4c2d555a46b1bf9113d76caee204e6d6bc8"} Nov 04 01:21:49.060377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:49.060281 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:49.060648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:49.060617 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:49.840506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:49.840469 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:21:49.840998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:49.840974 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 04 01:21:49.934329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:49.933732 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" event={"ID":"01c41ff4-0264-4807-99e7-daa17e5d0f5a","Type":"ContainerStarted","Data":"5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21"} Nov 04 01:21:49.958015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:49.957952 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podStartSLOduration=2.957933533 podStartE2EDuration="2.957933533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:21:49.953573438 +0000 UTC m=+1448.925322310" watchObservedRunningTime="2026-11-04 01:21:49.957933533 +0000 UTC m=+1448.929682420" Nov 04 01:21:50.060087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:50.060048 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.060314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:50.060110 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:50.282310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:50.282268 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:50.845147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:50.840904 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:21:50.845147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:50.840980 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 04 01:21:51.063197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:51.062495 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.063197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:51.062564 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:51.853813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:51.853775 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 04 01:21:51.854429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:51.854386 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 04 01:21:51.927145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:51.926813 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:52.061553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:52.061511 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 01:21:52.061831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:52.061799 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 01:21:52.809734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:52.783689 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.103254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:52.980913 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:21:53.163146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:53.064918 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.163146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:53.139217 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:21:53.163146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:53.103320 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.386205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:53.382074 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.682207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:53.670945 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:53.841194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:53.809227 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.121147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.119052 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:54.121147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.119114 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.175154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.139696 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:21:54.250435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.179655 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.255636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.179688 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:21:54.270341 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.183234 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:54.270723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.270669 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.271499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.271469 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:21:54.307477 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.307424 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.319371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.319339 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:21:54.770147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.769267 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:54.793147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.774778 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:21:54.863149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.822108 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:21:54.863149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.822721 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:54.863149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.823584 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:21:54.863644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.863608 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:54.863839 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:54.863804 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.005155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.003226 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:55.005155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.003298 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.214202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.181290 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:55.214202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.193324 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.255930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.255881 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.266661 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.266613 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.392146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.314680 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.476455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.326237 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:55.476455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.468771 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.476455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.421490 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:55.476455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.468953 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.801146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.769923 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.825000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.824958 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:55.825359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.825328 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:21:55.830268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:55.830235 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:21:56.219548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.219450 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:21:56.224241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.224189 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.283282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.283226 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.318146 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:21:56.314960 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.107s" Nov 04 01:21:56.343145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.334606 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:56.343145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.334746 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:21:56.343145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.338739 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:56.348466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.348435 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:21:56.473234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.472661 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.870533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.643595 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.870533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.668452 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:56.870533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.771374 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.870533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.805272 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:56.870533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.844237 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:56.870533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:56.844591 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:21:57.261202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:57.225308 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.562205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:57.341280 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.562205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:57.348653 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:21:57.562205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:57.487193 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:21:57.880202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:57.867187 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:57.925451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:57.925389 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:21:58.030107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.030065 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:21:58.081159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.059973 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:21:58.542450 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.542386 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:21:58.718560 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:21:58.718523 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.178s" Nov 04 01:21:58.718899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.718878 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:58.719285 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.719267 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:21:58.719450 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.719436 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:21:58.719590 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.719576 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:21:58.719727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.719714 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:21:58.719887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.719872 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:21:58.722308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.721490 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:21:58.742729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.742702 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:58.746919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.746896 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:21:58.848980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:58.848893 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:21:59.037299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.037230 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp"] Nov 04 01:21:59.046420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.046388 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.084577 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.084646 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.084712 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.084819 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndwq\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-kube-api-access-2ndwq\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.084846 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.084890 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.084964 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.085047 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.128222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.085193 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.129094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.129050 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp"] Nov 04 01:21:59.143677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.140109 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl"] Nov 04 01:21:59.224699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.158430 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.224699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.181196 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl"] Nov 04 01:21:59.243654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.227335 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.245179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.245145 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.245521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.245497 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.245682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.245666 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.245818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.245802 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.245964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.245944 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfg7k\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-kube-api-access-qfg7k\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.246167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.246137 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.246373 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.246355 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.287000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.286964 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.287266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.287244 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.287437 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.287421 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndwq\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-kube-api-access-2ndwq\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.287564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.287549 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.289888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.286099 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.289888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.287739 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.289888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.285685 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.289888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.287905 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.289888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.287992 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.289888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.288031 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f8e10-c023-4ed5-a75a-fed43dbadae7-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.289888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.288068 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.289888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.288093 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.290807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.290758 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.292115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.292092 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.317334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.303722 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.354145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.352930 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndwq\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-kube-api-access-2ndwq\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.392201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.391605 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f8e10-c023-4ed5-a75a-fed43dbadae7-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.392201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.391661 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.392201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.391686 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.392201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.391747 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.392201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.391830 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.392201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.391940 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfg7k\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-kube-api-access-qfg7k\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.392201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.392036 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.392201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.392063 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.392847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.392109 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.407147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.406737 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-certs\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.408005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.407864 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.423177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.408431 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.423177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.408727 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f8e10-c023-4ed5-a75a-fed43dbadae7-istiod-ca-cert\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.423177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.409208 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.423177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.416738 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-data\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.423177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.419139 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd"] Nov 04 01:21:59.423177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.422610 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-envoy\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.423636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.423286 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-credential-socket\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.426943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.425894 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd"] Nov 04 01:21:59.426943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.426001 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.430018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.429963 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.430741 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.430696 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-podinfo\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.448200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.445717 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-token\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.448200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.446443 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfg7k\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-kube-api-access-qfg7k\") pod \"e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.448200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.446712 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:21:59.448200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.447455 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:21:59.448200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.447525 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:21:59.448200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.447659 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:21:59.509523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.508607 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.509523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.508731 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.509523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.508776 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjdk\" (UniqueName: \"kubernetes.io/projected/55f4aab3-be13-4af0-8b01-25843096202d-kube-api-access-tsjdk\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.528593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.528522 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:21:59.547194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.547140 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:21:59.610538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.610188 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.610538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.610253 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjdk\" (UniqueName: \"kubernetes.io/projected/55f4aab3-be13-4af0-8b01-25843096202d-kube-api-access-tsjdk\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.611275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.610507 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.612151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.612097 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.614807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.614750 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.643636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.643521 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjdk\" (UniqueName: \"kubernetes.io/projected/55f4aab3-be13-4af0-8b01-25843096202d-kube-api-access-tsjdk\") pod \"maas-api-revoke-sub-688f431e-cmjkd\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.842378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.842338 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:21:59.861781 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:21:59.861750 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528f8e10_c023_4ed5_a75a_fed43dbadae7.slice/crio-8585806518af2ac7a4103a9ca7d945b0713386cab357e6688e944452541ef716 WatchSource:0}: Error finding container 8585806518af2ac7a4103a9ca7d945b0713386cab357e6688e944452541ef716: Status 404 returned error can't find the container with id 8585806518af2ac7a4103a9ca7d945b0713386cab357e6688e944452541ef716 Nov 04 01:21:59.865463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.864577 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:59.865463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.864668 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:59.865463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.864714 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:59.871989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.871951 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl"] Nov 04 01:21:59.890983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.890909 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp"] Nov 04 01:21:59.899463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.899420 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:59.899677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.899656 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:21:59.899826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:21:59.899800 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:00.184219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.184111 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd"] Nov 04 01:22:00.362896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.362859 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" event={"ID":"55f4aab3-be13-4af0-8b01-25843096202d","Type":"ContainerStarted","Data":"2df28c32b9d349285fb7783c498773f1d00b20d4d494de08c32cf0d5c827e0c7"} Nov 04 01:22:00.366487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.365324 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" event={"ID":"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa","Type":"ContainerStarted","Data":"ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d"} Nov 04 01:22:00.366487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.365374 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" event={"ID":"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa","Type":"ContainerStarted","Data":"6c547a309da32ba193e68bc2b6e5f44d6bc1ffe1267d78e79ea326abbaeddedb"} Nov 04 01:22:00.370095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.370064 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" event={"ID":"528f8e10-c023-4ed5-a75a-fed43dbadae7","Type":"ContainerStarted","Data":"d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae"} Nov 04 01:22:00.370221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.370113 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" event={"ID":"528f8e10-c023-4ed5-a75a-fed43dbadae7","Type":"ContainerStarted","Data":"8585806518af2ac7a4103a9ca7d945b0713386cab357e6688e944452541ef716"} Nov 04 01:22:00.395698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.395288 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podStartSLOduration=3.395269444 podStartE2EDuration="3.395269444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:00.392939557 +0000 UTC m=+1459.364688430" watchObservedRunningTime="2026-11-04 01:22:00.395269444 +0000 UTC m=+1459.367018317" Nov 04 01:22:00.421792 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.421723 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podStartSLOduration=3.421701287 podStartE2EDuration="3.421701287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:00.415843155 +0000 UTC m=+1459.387592039" watchObservedRunningTime="2026-11-04 01:22:00.421701287 +0000 UTC m=+1459.393450159" Nov 04 01:22:00.528968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.528923 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:22:00.548309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:00.548239 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:22:01.419255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:01.419210 5051 generic.go:356] "Generic (PLEG): container finished" podID="55f4aab3-be13-4af0-8b01-25843096202d" containerID="afc7798f538e610ef92f877404fcee37f457f680af4906078f8eea012971f876" exitCode=0 Nov 04 01:22:01.419471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:01.419442 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" event={"ID":"55f4aab3-be13-4af0-8b01-25843096202d","Type":"ContainerDied","Data":"afc7798f538e610ef92f877404fcee37f457f680af4906078f8eea012971f876"} Nov 04 01:22:01.530342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:01.530305 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.530573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:01.530368 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:22:01.562411 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:01.562290 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:22:01.563048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:01.562999 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 04 01:22:02.534695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:02.534110 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.534695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:02.534194 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:22:02.557145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:02.555526 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:22:02.557145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:02.555583 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 04 01:22:03.094525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.094489 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:22:03.170158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.162601 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-xks-ca\") pod \"55f4aab3-be13-4af0-8b01-25843096202d\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " Nov 04 01:22:03.170158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.162759 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-openshift-ca\") pod \"55f4aab3-be13-4af0-8b01-25843096202d\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " Nov 04 01:22:03.170158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.162988 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f4aab3-be13-4af0-8b01-25843096202d-kube-api-access-tsjdk\" (UniqueName: \"kubernetes.io/projected/55f4aab3-be13-4af0-8b01-25843096202d-kube-api-access-tsjdk\") pod \"55f4aab3-be13-4af0-8b01-25843096202d\" (UID: \"55f4aab3-be13-4af0-8b01-25843096202d\") " Nov 04 01:22:03.171034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.170996 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-openshift-ca" pod "55f4aab3-be13-4af0-8b01-25843096202d" (UID: "55f4aab3-be13-4af0-8b01-25843096202d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:03.232722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.232056 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f4aab3-be13-4af0-8b01-25843096202d-kube-api-access-tsjdk" pod "55f4aab3-be13-4af0-8b01-25843096202d" (UID: "55f4aab3-be13-4af0-8b01-25843096202d"). InnerVolumeSpecName "kube-api-access-tsjdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:03.233045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.233008 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-xks-ca" pod "55f4aab3-be13-4af0-8b01-25843096202d" (UID: "55f4aab3-be13-4af0-8b01-25843096202d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:03.286142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.276781 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjdk\" (UniqueName: \"kubernetes.io/projected/55f4aab3-be13-4af0-8b01-25843096202d-kube-api-access-tsjdk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:03.286142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.276818 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:03.286142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.276836 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f4aab3-be13-4af0-8b01-25843096202d-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:03.439057 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.438782 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" Nov 04 01:22:03.439057 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.438811 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd" event={"ID":"55f4aab3-be13-4af0-8b01-25843096202d","Type":"ContainerDied","Data":"2df28c32b9d349285fb7783c498773f1d00b20d4d494de08c32cf0d5c827e0c7"} Nov 04 01:22:03.439057 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.438896 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2df28c32b9d349285fb7783c498773f1d00b20d4d494de08c32cf0d5c827e0c7" Nov 04 01:22:03.530131 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.530031 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.530131 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.530097 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:22:03.541314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.541281 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn"] Nov 04 01:22:03.542005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.541983 5051 cpu_manager.go:414] "containerMap: removing container" podUID="55f4aab3-be13-4af0-8b01-25843096202d" containerName="revoke-keys" Nov 04 01:22:03.542005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.542005 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f4aab3-be13-4af0-8b01-25843096202d" containerName="revoke-keys" Nov 04 01:22:03.543684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.542264 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f4aab3-be13-4af0-8b01-25843096202d" containerName="revoke-keys" Nov 04 01:22:03.558150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.548259 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:22:03.558150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.548335 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 04 01:22:03.592481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.592440 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.599672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.599550 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:22:03.607213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.603534 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:22:03.607439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.603622 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:22:03.607523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.603703 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:22:03.705836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.705790 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn"] Nov 04 01:22:03.736941 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.736900 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.737191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.737033 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9trr\" (UniqueName: \"kubernetes.io/projected/3cef703d-c353-45d3-94f7-b0f5a460e27a-kube-api-access-p9trr\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.737191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.737066 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.840193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.839211 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9trr\" (UniqueName: \"kubernetes.io/projected/3cef703d-c353-45d3-94f7-b0f5a460e27a-kube-api-access-p9trr\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.840193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.839263 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.840193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.839522 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.840479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.840445 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.845184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.842726 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:03.910159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:03.909716 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9trr\" (UniqueName: \"kubernetes.io/projected/3cef703d-c353-45d3-94f7-b0f5a460e27a-kube-api-access-p9trr\") pod \"maas-api-revoke-sub-3bc2f54f-dtsvn\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:04.030429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:04.030397 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:04.149820 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:22:04.149749 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:22:04.510178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:04.508981 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn"] Nov 04 01:22:04.510292 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:04.510233 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cef703d_c353_45d3_94f7_b0f5a460e27a.slice/crio-6d13a0fd4c20a988fb4307ca678b11c2adf111ae3622e67d525fb0a576d885b2 WatchSource:0}: Error finding container 6d13a0fd4c20a988fb4307ca678b11c2adf111ae3622e67d525fb0a576d885b2: Status 404 returned error can't find the container with id 6d13a0fd4c20a988fb4307ca678b11c2adf111ae3622e67d525fb0a576d885b2 Nov 04 01:22:04.528928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:04.528895 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.529036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:04.528959 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:22:04.547423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:04.547395 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:22:04.547533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:04.547446 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 04 01:22:05.546142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:05.544575 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.546142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:05.544637 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:22:05.626146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:05.622051 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:22:05.626146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:05.622113 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 04 01:22:05.711215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:05.699745 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" event={"ID":"3cef703d-c353-45d3-94f7-b0f5a460e27a","Type":"ContainerStarted","Data":"5a1180f3c4514c36dac4d3e45353fc33f9acedab38fa166657a4ee545ed2ec6b"} Nov 04 01:22:05.743290 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:05.743247 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" event={"ID":"3cef703d-c353-45d3-94f7-b0f5a460e27a","Type":"ContainerStarted","Data":"6d13a0fd4c20a988fb4307ca678b11c2adf111ae3622e67d525fb0a576d885b2"} Nov 04 01:22:05.825204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:05.824099 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" podStartSLOduration=2.824077539 podStartE2EDuration="2.824077539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:05.770998554 +0000 UTC m=+1464.742747427" watchObservedRunningTime="2026-11-04 01:22:05.824077539 +0000 UTC m=+1464.795826410" Nov 04 01:22:05.941769 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:22:05.941223 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cef703d_c353_45d3_94f7_b0f5a460e27a.slice/crio-conmon-5a1180f3c4514c36dac4d3e45353fc33f9acedab38fa166657a4ee545ed2ec6b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cef703d_c353_45d3_94f7_b0f5a460e27a.slice/crio-5a1180f3c4514c36dac4d3e45353fc33f9acedab38fa166657a4ee545ed2ec6b.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:22:06.537150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:06.529546 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.537150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:06.529613 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:22:06.549794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:06.549757 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:22:06.550448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:06.550414 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 04 01:22:06.716845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:06.716804 5051 generic.go:356] "Generic (PLEG): container finished" podID="3cef703d-c353-45d3-94f7-b0f5a460e27a" containerID="5a1180f3c4514c36dac4d3e45353fc33f9acedab38fa166657a4ee545ed2ec6b" exitCode=0 Nov 04 01:22:06.717065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:06.716876 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" event={"ID":"3cef703d-c353-45d3-94f7-b0f5a460e27a","Type":"ContainerDied","Data":"5a1180f3c4514c36dac4d3e45353fc33f9acedab38fa166657a4ee545ed2ec6b"} Nov 04 01:22:07.143012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.142746 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p"] Nov 04 01:22:07.147770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.147735 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p"] Nov 04 01:22:07.147973 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.147840 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.152823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.152794 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"distinct-2-w2-d7e45d-kserve-self-signed-certs\"" Nov 04 01:22:07.152962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.152942 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"openshift-service-ca.crt\"" Nov 04 01:22:07.154068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.153531 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"kube-root-ca.crt\"" Nov 04 01:22:07.154068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.153623 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d7e45d\"/\"default-dockercfg-9sgp7\"" Nov 04 01:22:07.265144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.262725 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b70f01-982e-49b9-84d3-76efa67498e9-tls-certs\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.265144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.262795 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-home\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.265144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.262844 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-dshm\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.265144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.262870 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-tmp-dir\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.265144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.263008 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gnv\" (UniqueName: \"kubernetes.io/projected/02b70f01-982e-49b9-84d3-76efa67498e9-kube-api-access-l6gnv\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.265144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.263174 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-model-cache\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.364849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.364800 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-model-cache\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.365173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.365151 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b70f01-982e-49b9-84d3-76efa67498e9-tls-certs\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.365361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.365337 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-home\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.365519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.365496 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-dshm\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.365626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.365610 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-tmp-dir\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.365911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.365894 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gnv\" (UniqueName: \"kubernetes.io/projected/02b70f01-982e-49b9-84d3-76efa67498e9-kube-api-access-l6gnv\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.367481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.367385 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-tmp-dir\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.367481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.367419 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-model-cache\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.368000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.367982 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-home\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.389152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.385854 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b70f01-982e-49b9-84d3-76efa67498e9-tls-certs\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.389152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.387718 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-dshm\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.395164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.394309 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gnv\" (UniqueName: \"kubernetes.io/projected/02b70f01-982e-49b9-84d3-76efa67498e9-kube-api-access-l6gnv\") pod \"distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.466112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.466077 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:07.529430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.529345 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.529430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.529405 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 04 01:22:07.554161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.551891 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 04 01:22:07.554161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.551936 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 04 01:22:07.814704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.814393 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p"] Nov 04 01:22:07.988103 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:07.988077 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:08.081979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.081272 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cef703d-c353-45d3-94f7-b0f5a460e27a-kube-api-access-p9trr\" (UniqueName: \"kubernetes.io/projected/3cef703d-c353-45d3-94f7-b0f5a460e27a-kube-api-access-p9trr\") pod \"3cef703d-c353-45d3-94f7-b0f5a460e27a\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " Nov 04 01:22:08.081979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.081515 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-xks-ca\") pod \"3cef703d-c353-45d3-94f7-b0f5a460e27a\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " Nov 04 01:22:08.081979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.081676 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-openshift-ca\") pod \"3cef703d-c353-45d3-94f7-b0f5a460e27a\" (UID: \"3cef703d-c353-45d3-94f7-b0f5a460e27a\") " Nov 04 01:22:08.083706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.083434 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-openshift-ca" pod "3cef703d-c353-45d3-94f7-b0f5a460e27a" (UID: "3cef703d-c353-45d3-94f7-b0f5a460e27a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:08.092655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.092610 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cef703d-c353-45d3-94f7-b0f5a460e27a-kube-api-access-p9trr" pod "3cef703d-c353-45d3-94f7-b0f5a460e27a" (UID: "3cef703d-c353-45d3-94f7-b0f5a460e27a"). InnerVolumeSpecName "kube-api-access-p9trr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:08.094879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.094840 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-xks-ca" pod "3cef703d-c353-45d3-94f7-b0f5a460e27a" (UID: "3cef703d-c353-45d3-94f7-b0f5a460e27a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:08.185405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.185315 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:08.185405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.185351 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cef703d-c353-45d3-94f7-b0f5a460e27a-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:08.185405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.185366 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9trr\" (UniqueName: \"kubernetes.io/projected/3cef703d-c353-45d3-94f7-b0f5a460e27a-kube-api-access-p9trr\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:08.743422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.743308 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" event={"ID":"02b70f01-982e-49b9-84d3-76efa67498e9","Type":"ContainerStarted","Data":"a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9"} Nov 04 01:22:08.743422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.743364 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" event={"ID":"02b70f01-982e-49b9-84d3-76efa67498e9","Type":"ContainerStarted","Data":"5f80715ee856a311f2995616539caf9b65d7b292ae888e433345fcc85e27a864"} Nov 04 01:22:08.746172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.746146 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" Nov 04 01:22:08.746296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.746215 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn" event={"ID":"3cef703d-c353-45d3-94f7-b0f5a460e27a","Type":"ContainerDied","Data":"6d13a0fd4c20a988fb4307ca678b11c2adf111ae3622e67d525fb0a576d885b2"} Nov 04 01:22:08.746296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.746260 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d13a0fd4c20a988fb4307ca678b11c2adf111ae3622e67d525fb0a576d885b2" Nov 04 01:22:08.766697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.766633 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podStartSLOduration=1.766612565 podStartE2EDuration="1.766612565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:08.759820073 +0000 UTC m=+1467.731568945" watchObservedRunningTime="2026-11-04 01:22:08.766612565 +0000 UTC m=+1467.738361448" Nov 04 01:22:08.795577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.795532 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:22:08.796382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.796347 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:22:08.797167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.797108 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:22:08.938427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.937464 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:22:08.938977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.938950 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:22:08.940208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:08.940181 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:22:09.381903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.381859 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc"] Nov 04 01:22:09.382571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.382554 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3cef703d-c353-45d3-94f7-b0f5a460e27a" containerName="revoke-keys" Nov 04 01:22:09.382661 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.382573 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cef703d-c353-45d3-94f7-b0f5a460e27a" containerName="revoke-keys" Nov 04 01:22:09.382723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.382686 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cef703d-c353-45d3-94f7-b0f5a460e27a" containerName="revoke-keys" Nov 04 01:22:09.386664 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.386644 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.393997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.393956 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:22:09.394161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.394023 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:22:09.394161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.394022 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:22:09.394161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.394132 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:22:09.395774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.395740 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc"] Nov 04 01:22:09.421127 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.421081 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.421304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.421159 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.421304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.421292 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrq5\" (UniqueName: \"kubernetes.io/projected/7fb8a4ec-5f08-48f2-98cd-865a6face69a-kube-api-access-fsrq5\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.467210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.467177 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:09.523008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.522970 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.523223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.523020 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.523223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.523077 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrq5\" (UniqueName: \"kubernetes.io/projected/7fb8a4ec-5f08-48f2-98cd-865a6face69a-kube-api-access-fsrq5\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.523867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.523843 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.525674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.525604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.532360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.532331 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrq5\" (UniqueName: \"kubernetes.io/projected/7fb8a4ec-5f08-48f2-98cd-865a6face69a-kube-api-access-fsrq5\") pod \"maas-api-revoke-sub-0370b5c7-vswkc\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.708584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.708495 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:09.998392 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:09.998347 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb8a4ec_5f08_48f2_98cd_865a6face69a.slice/crio-459763750b1d22d74d5e4fd3fdbb65204e7456a7db0249374b971ce02e890b54 WatchSource:0}: Error finding container 459763750b1d22d74d5e4fd3fdbb65204e7456a7db0249374b971ce02e890b54: Status 404 returned error can't find the container with id 459763750b1d22d74d5e4fd3fdbb65204e7456a7db0249374b971ce02e890b54 Nov 04 01:22:09.999173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:09.999094 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc"] Nov 04 01:22:10.768711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:10.768671 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" event={"ID":"7fb8a4ec-5f08-48f2-98cd-865a6face69a","Type":"ContainerStarted","Data":"ed92b74ca2197ccce3a22e9c7673a124da098563d3a3ab27cb36bc477bd17a43"} Nov 04 01:22:10.768877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:10.768770 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" event={"ID":"7fb8a4ec-5f08-48f2-98cd-865a6face69a","Type":"ContainerStarted","Data":"459763750b1d22d74d5e4fd3fdbb65204e7456a7db0249374b971ce02e890b54"} Nov 04 01:22:10.847546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:10.843048 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" podStartSLOduration=1.843023047 podStartE2EDuration="1.843023047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:10.809371132 +0000 UTC m=+1469.781120004" watchObservedRunningTime="2026-11-04 01:22:10.843023047 +0000 UTC m=+1469.814771920" Nov 04 01:22:11.781454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:11.781411 5051 generic.go:356] "Generic (PLEG): container finished" podID="7fb8a4ec-5f08-48f2-98cd-865a6face69a" containerID="ed92b74ca2197ccce3a22e9c7673a124da098563d3a3ab27cb36bc477bd17a43" exitCode=0 Nov 04 01:22:11.781454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:11.781448 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" event={"ID":"7fb8a4ec-5f08-48f2-98cd-865a6face69a","Type":"ContainerDied","Data":"ed92b74ca2197ccce3a22e9c7673a124da098563d3a3ab27cb36bc477bd17a43"} Nov 04 01:22:12.242395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.242363 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj"] Nov 04 01:22:12.251922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.251891 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.258829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.258794 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj"] Nov 04 01:22:12.345037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345002 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.345237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345071 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.345237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345156 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpql\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-kube-api-access-9zpql\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.345237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345223 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.345394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345371 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb01ae-97a9-4691-8f8b-4573f186cc87-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.345438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345413 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.345438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345434 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.345592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345577 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.345648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.345610 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447245 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447397 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb01ae-97a9-4691-8f8b-4573f186cc87-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447447 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447474 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447581 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447631 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447739 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447821 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.447852 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpql\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-kube-api-access-9zpql\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.448829 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.449618 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb01ae-97a9-4691-8f8b-4573f186cc87-istiod-ca-cert\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.451887 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-data\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.453046 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-certs\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.454144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.452226 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-credential-socket\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.475486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.475414 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-envoy\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.510783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.510701 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpql\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-kube-api-access-9zpql\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.511372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.511345 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-token\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.511471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.511379 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-podinfo\") pod \"e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.573670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.573629 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:12.707614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.705465 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf"] Nov 04 01:22:12.716306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.716274 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf"] Nov 04 01:22:12.716468 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.716386 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.769839 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774544 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.774519 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.774557 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.774592 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.774636 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwv9m\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-kube-api-access-cwv9m\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.774735 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.774855 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.774913 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab06cee-b976-406b-a5b6-1073613a5bad-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.774985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.774970 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.877989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.877950 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878017 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878041 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878069 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878110 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwv9m\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-kube-api-access-cwv9m\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878210 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878395 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878450 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab06cee-b976-406b-a5b6-1073613a5bad-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878500 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878558 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj"] Nov 04 01:22:12.878962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878920 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.878962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.878929 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.879316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.879293 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.879646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.879618 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.879718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.879649 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab06cee-b976-406b-a5b6-1073613a5bad-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.881601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.881574 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.881919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.881902 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.887947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.887838 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.887947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.887860 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwv9m\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-kube-api-access-cwv9m\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:12.890306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.890265 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:12.890388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.890358 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:12.890455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.890406 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:12.999793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:12.999769 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:13.077586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.077500 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:13.091325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.091298 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-openshift-ca\") pod \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " Nov 04 01:22:13.091851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.091731 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-openshift-ca" pod "7fb8a4ec-5f08-48f2-98cd-865a6face69a" (UID: "7fb8a4ec-5f08-48f2-98cd-865a6face69a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:13.092309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.091999 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-xks-ca\") pod \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " Nov 04 01:22:13.092531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.092516 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb8a4ec-5f08-48f2-98cd-865a6face69a-kube-api-access-fsrq5\" (UniqueName: \"kubernetes.io/projected/7fb8a4ec-5f08-48f2-98cd-865a6face69a-kube-api-access-fsrq5\") pod \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\" (UID: \"7fb8a4ec-5f08-48f2-98cd-865a6face69a\") " Nov 04 01:22:13.093944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.093923 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:13.097516 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.097488 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-xks-ca" pod "7fb8a4ec-5f08-48f2-98cd-865a6face69a" (UID: "7fb8a4ec-5f08-48f2-98cd-865a6face69a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:13.100134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.100099 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb8a4ec-5f08-48f2-98cd-865a6face69a-kube-api-access-fsrq5" pod "7fb8a4ec-5f08-48f2-98cd-865a6face69a" (UID: "7fb8a4ec-5f08-48f2-98cd-865a6face69a"). InnerVolumeSpecName "kube-api-access-fsrq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:13.247644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.246645 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb8a4ec-5f08-48f2-98cd-865a6face69a-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:13.247644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.246685 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsrq5\" (UniqueName: \"kubernetes.io/projected/7fb8a4ec-5f08-48f2-98cd-865a6face69a-kube-api-access-fsrq5\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:13.402456 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:13.402088 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab06cee_b976_406b_a5b6_1073613a5bad.slice/crio-78a989093ed1adc8b6d0bd8e4e8ad7cb10a386353e618034d96790cae33b246d WatchSource:0}: Error finding container 78a989093ed1adc8b6d0bd8e4e8ad7cb10a386353e618034d96790cae33b246d: Status 404 returned error can't find the container with id 78a989093ed1adc8b6d0bd8e4e8ad7cb10a386353e618034d96790cae33b246d Nov 04 01:22:13.403747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.403390 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf"] Nov 04 01:22:13.406896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.406352 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:13.406896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.406442 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:13.406896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.406486 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:13.815382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.815352 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" Nov 04 01:22:13.815656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.815363 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc" event={"ID":"7fb8a4ec-5f08-48f2-98cd-865a6face69a","Type":"ContainerDied","Data":"459763750b1d22d74d5e4fd3fdbb65204e7456a7db0249374b971ce02e890b54"} Nov 04 01:22:13.815736 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.815683 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="459763750b1d22d74d5e4fd3fdbb65204e7456a7db0249374b971ce02e890b54" Nov 04 01:22:13.817473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.817446 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" event={"ID":"9ab06cee-b976-406b-a5b6-1073613a5bad","Type":"ContainerStarted","Data":"90d0a5039e4a9f5eba4f4c8719046e15417cdc2a1c9d8247ef616321af104fa2"} Nov 04 01:22:13.817668 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.817649 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" event={"ID":"9ab06cee-b976-406b-a5b6-1073613a5bad","Type":"ContainerStarted","Data":"78a989093ed1adc8b6d0bd8e4e8ad7cb10a386353e618034d96790cae33b246d"} Nov 04 01:22:13.820681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.820655 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" event={"ID":"56eb01ae-97a9-4691-8f8b-4573f186cc87","Type":"ContainerStarted","Data":"a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c"} Nov 04 01:22:13.820803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.820700 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" event={"ID":"56eb01ae-97a9-4691-8f8b-4573f186cc87","Type":"ContainerStarted","Data":"f0bca35da8abf06522b7d3f987a6d78d34c631984cc59b40c0fad243286289a5"} Nov 04 01:22:13.846918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:13.846861 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podStartSLOduration=1.846844334 podStartE2EDuration="1.846844334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:13.842238461 +0000 UTC m=+1472.813987345" watchObservedRunningTime="2026-11-04 01:22:13.846844334 +0000 UTC m=+1472.818593207" Nov 04 01:22:14.080845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:14.080756 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:14.081500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:14.081477 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:22:14.081648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:14.081625 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 04 01:22:14.578146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:14.576872 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:14.578146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:14.577013 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:22:14.578146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:14.577061 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:22:15.085143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:15.084506 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:22:15.085143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:15.084574 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 04 01:22:15.580145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:15.578529 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:22:15.580145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:15.578591 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:22:16.079145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:16.078482 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:22:16.079145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:16.078546 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 04 01:22:16.582144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:16.578724 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:22:16.582144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:16.578799 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:22:17.113332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:17.113293 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:22:17.113632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:17.113598 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 04 01:22:17.476210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:17.475443 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:17.542235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:17.542204 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:17.586142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:17.579540 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:22:17.586142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:17.579600 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:22:17.607145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:17.602986 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podStartSLOduration=5.602966339 podStartE2EDuration="5.602966339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:13.884626119 +0000 UTC m=+1472.856374993" watchObservedRunningTime="2026-11-04 01:22:17.602966339 +0000 UTC m=+1476.574715211" Nov 04 01:22:18.000174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:17.996583 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:18.092150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.081525 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:22:18.092150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.081584 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 04 01:22:18.526773 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.526738 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9"] Nov 04 01:22:18.527446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.527431 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7fb8a4ec-5f08-48f2-98cd-865a6face69a" containerName="revoke-keys" Nov 04 01:22:18.527510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.527447 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb8a4ec-5f08-48f2-98cd-865a6face69a" containerName="revoke-keys" Nov 04 01:22:18.527600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.527590 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb8a4ec-5f08-48f2-98cd-865a6face69a" containerName="revoke-keys" Nov 04 01:22:18.532532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.532514 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.542180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.542157 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:22:18.542180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.542173 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:22:18.542301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.542201 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:22:18.542301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.542245 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:22:18.543223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.543198 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9"] Nov 04 01:22:18.586143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.582477 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.586143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.582549 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5267q\" (UniqueName: \"kubernetes.io/projected/5dd49052-d139-4365-b55c-9ea489e4922b-kube-api-access-5267q\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.586143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.582744 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.586143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.583217 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:22:18.586143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.583263 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:22:18.713814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.713779 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.714759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.714731 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5267q\" (UniqueName: \"kubernetes.io/projected/5dd49052-d139-4365-b55c-9ea489e4922b-kube-api-access-5267q\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.715199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.715176 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.716346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.716324 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.722837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.722813 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.734743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.734711 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5267q\" (UniqueName: \"kubernetes.io/projected/5dd49052-d139-4365-b55c-9ea489e4922b-kube-api-access-5267q\") pod \"maas-api-revoke-sub-e335f439-bv6l9\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:18.857206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:18.856812 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:19.084148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:19.082720 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:22:19.084148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:19.082777 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 04 01:22:19.396453 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:19.396380 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd49052_d139_4365_b55c_9ea489e4922b.slice/crio-a0d34ecb0243c3df9273abc9d93fed1d9cfda11bc6deea499167e84abb0db5ea WatchSource:0}: Error finding container a0d34ecb0243c3df9273abc9d93fed1d9cfda11bc6deea499167e84abb0db5ea: Status 404 returned error can't find the container with id a0d34ecb0243c3df9273abc9d93fed1d9cfda11bc6deea499167e84abb0db5ea Nov 04 01:22:19.400605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:19.399741 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9"] Nov 04 01:22:19.580976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:19.580483 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 04 01:22:19.580976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:19.580545 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 04 01:22:20.001222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:20.001185 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" event={"ID":"5dd49052-d139-4365-b55c-9ea489e4922b","Type":"ContainerStarted","Data":"f96873fe4379b7b2dc849f1d072d760e8d716672be902c88ae9c94a6bf15bd94"} Nov 04 01:22:20.001717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:20.001276 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" event={"ID":"5dd49052-d139-4365-b55c-9ea489e4922b","Type":"ContainerStarted","Data":"a0d34ecb0243c3df9273abc9d93fed1d9cfda11bc6deea499167e84abb0db5ea"} Nov 04 01:22:20.066232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:20.066164 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" podStartSLOduration=2.066145149 podStartE2EDuration="2.066145149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:20.061554573 +0000 UTC m=+1479.033303444" watchObservedRunningTime="2026-11-04 01:22:20.066145149 +0000 UTC m=+1479.037894023" Nov 04 01:22:20.085516 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:20.085479 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:22:20.085853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:20.085822 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 04 01:22:21.012498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:21.012459 5051 generic.go:356] "Generic (PLEG): container finished" podID="5dd49052-d139-4365-b55c-9ea489e4922b" containerID="f96873fe4379b7b2dc849f1d072d760e8d716672be902c88ae9c94a6bf15bd94" exitCode=0 Nov 04 01:22:21.012915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:21.012548 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" event={"ID":"5dd49052-d139-4365-b55c-9ea489e4922b","Type":"ContainerDied","Data":"f96873fe4379b7b2dc849f1d072d760e8d716672be902c88ae9c94a6bf15bd94"} Nov 04 01:22:21.039928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:21.039895 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:21.041347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:21.041322 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:21.041609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:21.041590 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:22:21.078645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:21.078609 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 04 01:22:21.078917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:21.078887 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 04 01:22:22.083857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.083827 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:22.085160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.084374 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:22.087775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.087742 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:22:22.328302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.328277 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:22.435134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.435097 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd49052-d139-4365-b55c-9ea489e4922b-kube-api-access-5267q\" (UniqueName: \"kubernetes.io/projected/5dd49052-d139-4365-b55c-9ea489e4922b-kube-api-access-5267q\") pod \"5dd49052-d139-4365-b55c-9ea489e4922b\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " Nov 04 01:22:22.435343 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.435331 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-openshift-ca\") pod \"5dd49052-d139-4365-b55c-9ea489e4922b\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " Nov 04 01:22:22.435411 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.435397 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-xks-ca\") pod \"5dd49052-d139-4365-b55c-9ea489e4922b\" (UID: \"5dd49052-d139-4365-b55c-9ea489e4922b\") " Nov 04 01:22:22.435773 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.435738 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-openshift-ca" pod "5dd49052-d139-4365-b55c-9ea489e4922b" (UID: "5dd49052-d139-4365-b55c-9ea489e4922b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:22.435885 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.435871 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:22.437273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.437243 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd49052-d139-4365-b55c-9ea489e4922b-kube-api-access-5267q" pod "5dd49052-d139-4365-b55c-9ea489e4922b" (UID: "5dd49052-d139-4365-b55c-9ea489e4922b"). InnerVolumeSpecName "kube-api-access-5267q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:22.437474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.437455 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-xks-ca" pod "5dd49052-d139-4365-b55c-9ea489e4922b" (UID: "5dd49052-d139-4365-b55c-9ea489e4922b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:22.537611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.537531 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd49052-d139-4365-b55c-9ea489e4922b-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:22.537611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:22.537561 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5267q\" (UniqueName: \"kubernetes.io/projected/5dd49052-d139-4365-b55c-9ea489e4922b-kube-api-access-5267q\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:23.031010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:23.030984 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" Nov 04 01:22:23.031220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:23.031023 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9" event={"ID":"5dd49052-d139-4365-b55c-9ea489e4922b","Type":"ContainerDied","Data":"a0d34ecb0243c3df9273abc9d93fed1d9cfda11bc6deea499167e84abb0db5ea"} Nov 04 01:22:23.031220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:23.031070 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0d34ecb0243c3df9273abc9d93fed1d9cfda11bc6deea499167e84abb0db5ea" Nov 04 01:22:25.487737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:25.487694 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:25.895209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:25.894638 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.273148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:26.259208 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.651203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:26.650211 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.756145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:26.755646 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.850576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:26.849225 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.883092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:26.882417 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:26.904203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:26.903449 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:26.904377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:26.904236 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:22:26.963145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:26.957207 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:27.269206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:27.219407 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.547221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:27.487879 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.547221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:27.490432 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:27.668593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:27.668543 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.868203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:27.867068 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.868385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:27.868190 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.886151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:27.886089 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:27.906331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:27.906286 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:28.079146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.073233 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.248374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.185455 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:22:28.296388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.296352 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:22:28.334530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.334501 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:22:28.389284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.389252 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:22:28.389478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.389309 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:22:28.389478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.389347 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:28.488249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.488200 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.515213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.496690 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:22:28.529145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.501636 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.748146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.737867 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:22:28.748146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.742254 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:28.913205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.907201 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.913205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.910442 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:28.922149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.913955 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:28.922149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:28.914282 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:29.128351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.128306 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.204335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.204238 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:29.205588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.204542 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.348789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.289219 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.404276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.404227 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 04 01:22:29.526314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.525774 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:22:29.544076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.541825 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.823642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.823544 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.910831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.909488 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.912165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.911073 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:29.942275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.942236 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:29.942856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:29.942505 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:30.155203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:30.154210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.530090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:30.530043 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.629242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:30.629199 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.642301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:30.642260 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:30.855777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:30.855659 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:30.874259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:30.874207 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:30.874438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:30.874275 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:31.002447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.002400 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:31.004629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.004592 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:31.011870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.011835 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:22:31.019710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.015185 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:31.164340 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.164230 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:31.469200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.388920 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:31.549155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.548256 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:22:31.549155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.548624 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:22:31.630248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.630200 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:31.665593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.665086 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:31.665593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.665168 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:31.856224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:31.855906 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:32.051148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.036838 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:32.051148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.037299 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:32.051148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.037625 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:32.051148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.037916 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:32.302151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.294444 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:32.755207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.741170 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:32.788149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.782486 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:22:32.900250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.874268 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:33.002241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.987104 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2"] Nov 04 01:22:33.002241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.989714 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5dd49052-d139-4365-b55c-9ea489e4922b" containerName="revoke-keys" Nov 04 01:22:33.002241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.989738 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd49052-d139-4365-b55c-9ea489e4922b" containerName="revoke-keys" Nov 04 01:22:33.002241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:32.994721 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd49052-d139-4365-b55c-9ea489e4922b" containerName="revoke-keys" Nov 04 01:22:33.059341 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.051551 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:33.065165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.060621 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:33.075301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.075264 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:33.091693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.076296 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:33.363201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.353866 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:33.583301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.583270 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.618078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.617998 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:22:33.620236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.620213 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:22:33.651720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.633336 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:22:33.651720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.633427 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:22:33.678056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.677833 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.682878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.682813 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.683334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.683256 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsx9s\" (UniqueName: \"kubernetes.io/projected/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-kube-api-access-tsx9s\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.812790 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.797586 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:33.826094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.812485 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.826411 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.826384 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.826543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.826530 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsx9s\" (UniqueName: \"kubernetes.io/projected/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-kube-api-access-tsx9s\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.827265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.781713 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:33.848184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.841321 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.881804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.820912 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:33.882191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.875251 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:33.936140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.924833 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:22:33.936140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.924934 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2"] Nov 04 01:22:33.974008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.973979 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:22:33.999914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:33.999881 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsx9s\" (UniqueName: \"kubernetes.io/projected/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-kube-api-access-tsx9s\") pod \"maas-api-revoke-sub-ccedf22b-qdqp2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:34.022305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.022266 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:22:34.052585 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.052543 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:34.101192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.099567 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:22:34.101643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.101616 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:34.116837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.116776 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:22:34.142793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.142598 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:34.147087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.147064 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:34.208544 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.208516 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:22:34.209621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.209597 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:22:34.210955 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:22:34.210926 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:22:34.569768 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:34.569742 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2"] Nov 04 01:22:34.572142 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:34.572087 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c3b55d_f7da_4811_80f3_7b8451a6d3d2.slice/crio-33a9ce68c0009bba1549c6240b5e2837102c65faf07aae755671c82716552873 WatchSource:0}: Error finding container 33a9ce68c0009bba1549c6240b5e2837102c65faf07aae755671c82716552873: Status 404 returned error can't find the container with id 33a9ce68c0009bba1549c6240b5e2837102c65faf07aae755671c82716552873 Nov 04 01:22:35.040270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:35.040235 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" event={"ID":"11c3b55d-f7da-4811-80f3-7b8451a6d3d2","Type":"ContainerStarted","Data":"33a9ce68c0009bba1549c6240b5e2837102c65faf07aae755671c82716552873"} Nov 04 01:22:36.091697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:36.091602 5051 generic.go:356] "Generic (PLEG): container finished" podID="11c3b55d-f7da-4811-80f3-7b8451a6d3d2" containerID="5d1c9e59017c71fd0b0e925959a66689f8d9e4d984a1086da751c22b428e8701" exitCode=0 Nov 04 01:22:36.092809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:36.091694 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" event={"ID":"11c3b55d-f7da-4811-80f3-7b8451a6d3d2","Type":"ContainerDied","Data":"5d1c9e59017c71fd0b0e925959a66689f8d9e4d984a1086da751c22b428e8701"} Nov 04 01:22:37.280541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.280513 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:37.354455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.354359 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-kube-api-access-tsx9s\" (UniqueName: \"kubernetes.io/projected/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-kube-api-access-tsx9s\") pod \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " Nov 04 01:22:37.354455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.354427 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-openshift-ca\") pod \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " Nov 04 01:22:37.354717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.354699 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-xks-ca\") pod \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\" (UID: \"11c3b55d-f7da-4811-80f3-7b8451a6d3d2\") " Nov 04 01:22:37.354861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.354831 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-openshift-ca" pod "11c3b55d-f7da-4811-80f3-7b8451a6d3d2" (UID: "11c3b55d-f7da-4811-80f3-7b8451a6d3d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:37.355444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.355418 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:37.356896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.356863 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-kube-api-access-tsx9s" pod "11c3b55d-f7da-4811-80f3-7b8451a6d3d2" (UID: "11c3b55d-f7da-4811-80f3-7b8451a6d3d2"). InnerVolumeSpecName "kube-api-access-tsx9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:37.356993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.356952 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-xks-ca" pod "11c3b55d-f7da-4811-80f3-7b8451a6d3d2" (UID: "11c3b55d-f7da-4811-80f3-7b8451a6d3d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:37.456854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.456810 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:37.456854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:37.456852 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsx9s\" (UniqueName: \"kubernetes.io/projected/11c3b55d-f7da-4811-80f3-7b8451a6d3d2-kube-api-access-tsx9s\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:38.112392 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:38.112362 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" Nov 04 01:22:38.112579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:38.112363 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2" event={"ID":"11c3b55d-f7da-4811-80f3-7b8451a6d3d2","Type":"ContainerDied","Data":"33a9ce68c0009bba1549c6240b5e2837102c65faf07aae755671c82716552873"} Nov 04 01:22:38.112579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:38.112491 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a9ce68c0009bba1549c6240b5e2837102c65faf07aae755671c82716552873" Nov 04 01:22:39.736520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.736477 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv"] Nov 04 01:22:39.737194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.737179 5051 cpu_manager.go:414] "containerMap: removing container" podUID="11c3b55d-f7da-4811-80f3-7b8451a6d3d2" containerName="revoke-keys" Nov 04 01:22:39.737243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.737195 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c3b55d-f7da-4811-80f3-7b8451a6d3d2" containerName="revoke-keys" Nov 04 01:22:39.737315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.737305 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c3b55d-f7da-4811-80f3-7b8451a6d3d2" containerName="revoke-keys" Nov 04 01:22:39.740776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.740756 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.746259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.746235 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:22:39.746464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.746446 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:22:39.746577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.746562 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:22:39.746679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.746645 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:22:39.752530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.751671 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv"] Nov 04 01:22:39.821430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.821392 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.821626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.821440 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.821626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.821593 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc2d\" (UniqueName: \"kubernetes.io/projected/07cccfa6-abc9-44a4-9a84-762a19956113-kube-api-access-2mc2d\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.922994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.922953 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.923391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.923360 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.923520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.923459 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc2d\" (UniqueName: \"kubernetes.io/projected/07cccfa6-abc9-44a4-9a84-762a19956113-kube-api-access-2mc2d\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.923793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.923771 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.925835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.925809 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:39.931862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:39.931839 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc2d\" (UniqueName: \"kubernetes.io/projected/07cccfa6-abc9-44a4-9a84-762a19956113-kube-api-access-2mc2d\") pod \"maas-api-revoke-sub-7d3e62d0-b8dhv\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:40.077197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:40.073161 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:40.396060 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:40.396026 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cccfa6_abc9_44a4_9a84_762a19956113.slice/crio-b6ef1a779da9213e22fcc5138f57ef3856d99eacbc88a6fa70ef1d7043a80887 WatchSource:0}: Error finding container b6ef1a779da9213e22fcc5138f57ef3856d99eacbc88a6fa70ef1d7043a80887: Status 404 returned error can't find the container with id b6ef1a779da9213e22fcc5138f57ef3856d99eacbc88a6fa70ef1d7043a80887 Nov 04 01:22:40.396646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:40.396622 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv"] Nov 04 01:22:41.087524 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.087492 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt"] Nov 04 01:22:41.092342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.092306 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.103605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.103578 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt"] Nov 04 01:22:41.153395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.151957 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.153395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.152453 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbzps\" (UniqueName: \"kubernetes.io/projected/e9d72759-d28a-4941-a1dc-69c107b65b40-kube-api-access-fbzps\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.153395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.152633 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.165007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.164966 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" event={"ID":"07cccfa6-abc9-44a4-9a84-762a19956113","Type":"ContainerStarted","Data":"60e0e9402c7566f11a446dabdad8fc3c709ba34f8e265e6e1e7ab3c9122401c0"} Nov 04 01:22:41.165469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.165442 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" event={"ID":"07cccfa6-abc9-44a4-9a84-762a19956113","Type":"ContainerStarted","Data":"b6ef1a779da9213e22fcc5138f57ef3856d99eacbc88a6fa70ef1d7043a80887"} Nov 04 01:22:41.195019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.194963 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" podStartSLOduration=2.194948419 podStartE2EDuration="2.194948419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:41.189851914 +0000 UTC m=+1500.161600784" watchObservedRunningTime="2026-11-04 01:22:41.194948419 +0000 UTC m=+1500.166697343" Nov 04 01:22:41.253890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.253844 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.254319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.254274 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzps\" (UniqueName: \"kubernetes.io/projected/e9d72759-d28a-4941-a1dc-69c107b65b40-kube-api-access-fbzps\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.254584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.254495 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.254971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.254939 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.257226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.257204 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.264775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.264745 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzps\" (UniqueName: \"kubernetes.io/projected/e9d72759-d28a-4941-a1dc-69c107b65b40-kube-api-access-fbzps\") pod \"maas-api-revoke-sub-2a40b686-96gkt\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.401142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.401034 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:22:41.401142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.401034 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:22:41.411277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.411251 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:41.747507 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:41.747471 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d72759_d28a_4941_a1dc_69c107b65b40.slice/crio-41173ba50a23e313b0682dccd542c9bde4e8a09c55f29a46689e7438e7055105 WatchSource:0}: Error finding container 41173ba50a23e313b0682dccd542c9bde4e8a09c55f29a46689e7438e7055105: Status 404 returned error can't find the container with id 41173ba50a23e313b0682dccd542c9bde4e8a09c55f29a46689e7438e7055105 Nov 04 01:22:41.747940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:41.747918 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt"] Nov 04 01:22:42.176196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.176160 5051 generic.go:356] "Generic (PLEG): container finished" podID="07cccfa6-abc9-44a4-9a84-762a19956113" containerID="60e0e9402c7566f11a446dabdad8fc3c709ba34f8e265e6e1e7ab3c9122401c0" exitCode=0 Nov 04 01:22:42.176675 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.176246 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" event={"ID":"07cccfa6-abc9-44a4-9a84-762a19956113","Type":"ContainerDied","Data":"60e0e9402c7566f11a446dabdad8fc3c709ba34f8e265e6e1e7ab3c9122401c0"} Nov 04 01:22:42.178181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.178156 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" event={"ID":"e9d72759-d28a-4941-a1dc-69c107b65b40","Type":"ContainerStarted","Data":"c0e511ab0b1af42761b0b10f416c3f69cc50587b0d6c3dc5a4d4e0cbe78152eb"} Nov 04 01:22:42.178291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.178208 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" event={"ID":"e9d72759-d28a-4941-a1dc-69c107b65b40","Type":"ContainerStarted","Data":"41173ba50a23e313b0682dccd542c9bde4e8a09c55f29a46689e7438e7055105"} Nov 04 01:22:42.213960 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.213902 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" podStartSLOduration=1.213887012 podStartE2EDuration="1.213887012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:42.211421489 +0000 UTC m=+1501.183170361" watchObservedRunningTime="2026-11-04 01:22:42.213887012 +0000 UTC m=+1501.185635902" Nov 04 01:22:42.716026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.715991 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn"] Nov 04 01:22:42.720793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.720769 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.736114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.736078 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn"] Nov 04 01:22:42.753430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.753399 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4"] Nov 04 01:22:42.759759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.759726 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.784199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.784170 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4"] Nov 04 01:22:42.787589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.787559 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8fhh\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-kube-api-access-d8fhh\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.787722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.787687 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.787801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.787762 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5541dbc1-541d-4e63-ab47-5dd90fe99545-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.787871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.787846 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.787952 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.787939 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.788012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.788002 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.788074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.788039 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.788172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.788086 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.788172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.788109 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.814737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.814698 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx"] Nov 04 01:22:42.821874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.820744 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.823212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.823185 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs"] Nov 04 01:22:42.842403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.839223 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.842403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.841969 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx"] Nov 04 01:22:42.862210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.861215 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz"] Nov 04 01:22:42.873999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.873969 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:42.883529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.883497 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs"] Nov 04 01:22:42.889925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.889900 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.890187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.890109 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.890306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.890228 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8fhh\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-kube-api-access-d8fhh\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.904280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904239 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.904436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904323 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.904436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904362 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.904436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904417 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.904607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904465 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5541dbc1-541d-4e63-ab47-5dd90fe99545-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.904607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904506 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.904607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904549 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.904607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904588 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.904791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904680 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.904791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904707 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.904791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904745 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb95x\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-kube-api-access-mb95x\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.904938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904808 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.904938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904842 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.904938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904873 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.904938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904907 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.905191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904941 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.905191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.904992 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.905191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905085 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.905191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905157 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.905390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905202 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw8pt\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-kube-api-access-sw8pt\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.905390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905237 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.905390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905268 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf624c2c-5046-4423-8457-bf6e62aefad5-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.905390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905301 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhfz\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-kube-api-access-mwhfz\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.905390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905379 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.905637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905420 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.905692 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905667 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.905749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905720 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.905803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905751 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.905856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905806 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:42.905856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905847 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.906947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.905992 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.906947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.906288 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:42.911641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.911574 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5541dbc1-541d-4e63-ab47-5dd90fe99545-istiod-ca-cert\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.911911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.911891 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.912348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.912326 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-credential-socket\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.912563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.912547 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz"] Nov 04 01:22:42.912727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.912591 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-data\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.912850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.912822 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-certs\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.913099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.913076 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:42.946599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.928615 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-podinfo\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.946599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.930768 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7"] Nov 04 01:22:42.946599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.932098 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8fhh\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-kube-api-access-d8fhh\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.946599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.940305 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-envoy\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.946599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.942203 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:42.946599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.943579 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-token\") pod \"e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:42.972699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.969322 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7"] Nov 04 01:22:42.992229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.992195 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v"] Nov 04 01:22:42.998336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.998303 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v"] Nov 04 01:22:42.998497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:42.998459 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.014646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014609 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.014795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014670 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw8pt\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-kube-api-access-sw8pt\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.014795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014700 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.014795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014723 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf624c2c-5046-4423-8457-bf6e62aefad5-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.014795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014748 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhfz\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-kube-api-access-mwhfz\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.014795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014779 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.015063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014813 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl5pg\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-kube-api-access-dl5pg\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.015063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014851 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.015063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014887 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.015063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014940 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.015063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.014981 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.015595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.015565 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.015802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.015781 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf624c2c-5046-4423-8457-bf6e62aefad5-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.015877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.015858 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.015950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.015933 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.016283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016254 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.016377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016339 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.016458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016436 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.016516 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016482 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.016570 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016526 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.016570 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016555 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.016668 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016635 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.016726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016708 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.016898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016865 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.016972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016954 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa33d6-5854-4669-90db-6608e0e51278-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.017029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.016993 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.017090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017041 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.017090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017066 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.017213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017087 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.017415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017394 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.017494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017393 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.017667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017646 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.017729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017671 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.017932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017907 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.018007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017965 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbfz\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-kube-api-access-tdbfz\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.018007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017996 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.018131 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018054 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.018131 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018108 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.018234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018157 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.018234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018188 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.018234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018215 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.018383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018239 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.018383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018289 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.018383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.017996 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.018383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018352 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa33d6-5854-4669-90db-6608e0e51278-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.018861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018834 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.018943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018877 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.018943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018908 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.019054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.018943 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb95x\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-kube-api-access-mb95x\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.019054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.019000 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.019054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.019034 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.019252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.019059 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.019252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.019085 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.019252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.019144 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.019252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.019199 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.020886 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.020857 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.021166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.021144 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.021702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.021674 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.021989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.021966 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.022706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.022680 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.023238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.023214 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.023371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.023349 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.023860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.023835 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.025543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.025519 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.026923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.026905 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.029376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.029351 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw8pt\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-kube-api-access-sw8pt\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.029569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.029547 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.030637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.030579 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhfz\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-kube-api-access-mwhfz\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.032329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.032308 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.032419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.032399 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.035918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.035884 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.036320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.036300 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb95x\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-kube-api-access-mb95x\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.068067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.068000 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:43.084976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.084945 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120711 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120767 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120797 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120824 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120864 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120892 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120908 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120923 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120946 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa33d6-5854-4669-90db-6608e0e51278-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.120969 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121011 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054d98d6-5005-40a5-9243-8c0414ea2dbf-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121046 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121088 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbfz\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-kube-api-access-tdbfz\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121105 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121160 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121193 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcn8q\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-kube-api-access-rcn8q\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121224 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121239 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121257 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121306 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa33d6-5854-4669-90db-6608e0e51278-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121335 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121378 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121421 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121473 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121494 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl5pg\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-kube-api-access-dl5pg\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.121518 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.122177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.122189 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istiod-ca-cert\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.123785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.122512 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.123785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.122794 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-credential-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.123785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.122814 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-certs\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.123785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.123335 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.123785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.123571 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.123785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.123597 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-data\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.125139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.124546 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa33d6-5854-4669-90db-6608e0e51278-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.125139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.125076 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.125346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.125284 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-socket\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.129741 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.128695 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa33d6-5854-4669-90db-6608e0e51278-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.129741 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.129728 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-podinfo\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.129936 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.129891 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.132170 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.131133 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-envoy\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.133795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.133772 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.134553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.134525 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbfz\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-kube-api-access-tdbfz\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.135021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.134996 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-token\") pod \"e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.135212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.135192 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl5pg\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-kube-api-access-dl5pg\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.169898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.169496 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:43.175547 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.175426 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:43.207177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.205326 5051 generic.go:356] "Generic (PLEG): container finished" podID="e9d72759-d28a-4941-a1dc-69c107b65b40" containerID="c0e511ab0b1af42761b0b10f416c3f69cc50587b0d6c3dc5a4d4e0cbe78152eb" exitCode=0 Nov 04 01:22:43.207177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.205709 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" event={"ID":"e9d72759-d28a-4941-a1dc-69c107b65b40","Type":"ContainerDied","Data":"c0e511ab0b1af42761b0b10f416c3f69cc50587b0d6c3dc5a4d4e0cbe78152eb"} Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.222856 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.222942 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.223038 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.223083 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.223150 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054d98d6-5005-40a5-9243-8c0414ea2dbf-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.223195 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.223273 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.223324 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcn8q\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-kube-api-access-rcn8q\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.223480 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.225095 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.225388 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.227015 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.227871 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054d98d6-5005-40a5-9243-8c0414ea2dbf-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.228436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.228179 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.229732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.229557 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.235208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.234991 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.239914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.239855 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcn8q\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-kube-api-access-rcn8q\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.249091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.248299 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.252712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.252475 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:43.304843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.304676 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:43.322539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.318112 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:43.389006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.388849 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn"] Nov 04 01:22:43.399872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.398946 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4"] Nov 04 01:22:43.406582 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:43.406435 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5541dbc1_541d_4e63_ab47_5dd90fe99545.slice/crio-1cd3de94ea8674a501c44f4446788b1507e491934e548bea926b5f5420f67fc4 WatchSource:0}: Error finding container 1cd3de94ea8674a501c44f4446788b1507e491934e548bea926b5f5420f67fc4: Status 404 returned error can't find the container with id 1cd3de94ea8674a501c44f4446788b1507e491934e548bea926b5f5420f67fc4 Nov 04 01:22:43.423545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.421954 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.423545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.422069 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.423545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.422181 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.424115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.424076 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.424277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.424182 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.424277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.424234 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.646942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.646916 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:43.775134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.775044 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.775482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.775456 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.775604 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.775587 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.779989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.777857 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07cccfa6-abc9-44a4-9a84-762a19956113-kube-api-access-2mc2d\" (UniqueName: \"kubernetes.io/projected/07cccfa6-abc9-44a4-9a84-762a19956113-kube-api-access-2mc2d\") pod \"07cccfa6-abc9-44a4-9a84-762a19956113\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " Nov 04 01:22:43.779989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.778023 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-openshift-ca\") pod \"07cccfa6-abc9-44a4-9a84-762a19956113\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " Nov 04 01:22:43.779989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.778317 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca\") pod \"07cccfa6-abc9-44a4-9a84-762a19956113\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " Nov 04 01:22:43.799254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.799192 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-openshift-ca" pod "07cccfa6-abc9-44a4-9a84-762a19956113" (UID: "07cccfa6-abc9-44a4-9a84-762a19956113"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:43.803620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.803573 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.806310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.806267 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.806539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.806508 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.866358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.866317 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs"] Nov 04 01:22:43.905306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.900221 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx"] Nov 04 01:22:43.907912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.907880 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz"] Nov 04 01:22:43.911623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.911543 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca" pod "07cccfa6-abc9-44a4-9a84-762a19956113" (UID: "07cccfa6-abc9-44a4-9a84-762a19956113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:43.911939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.911909 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca\") pod \"07cccfa6-abc9-44a4-9a84-762a19956113\" (UID: \"07cccfa6-abc9-44a4-9a84-762a19956113\") " Nov 04 01:22:43.912064 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:22:43.912050 5051 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/07cccfa6-abc9-44a4-9a84-762a19956113/volumes/kubernetes.io~secret/maas-api-xks-ca Nov 04 01:22:43.912135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.912069 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca" pod "07cccfa6-abc9-44a4-9a84-762a19956113" (UID: "07cccfa6-abc9-44a4-9a84-762a19956113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:43.912432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.912403 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cccfa6-abc9-44a4-9a84-762a19956113-kube-api-access-2mc2d" pod "07cccfa6-abc9-44a4-9a84-762a19956113" (UID: "07cccfa6-abc9-44a4-9a84-762a19956113"). InnerVolumeSpecName "kube-api-access-2mc2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:43.916271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.916231 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mc2d\" (UniqueName: \"kubernetes.io/projected/07cccfa6-abc9-44a4-9a84-762a19956113-kube-api-access-2mc2d\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:43.916271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.916260 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:43.916406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.916292 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07cccfa6-abc9-44a4-9a84-762a19956113-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:43.944750 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.944583 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.945028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.944995 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:43.945186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:43.945165 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:44.070294 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.070246 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v"] Nov 04 01:22:44.102553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.101971 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:44.102553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.102076 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:44.102865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.102808 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:44.233220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.233164 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:44.234016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.233784 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:44.234016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.233885 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:22:44.273727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.258216 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7"] Nov 04 01:22:44.438989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.438939 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" event={"ID":"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf","Type":"ContainerStarted","Data":"73c3d9e2d9bb71e6865da33db5183ddfa73cc26f83b2cc318d1cd0be6f1dc7b9"} Nov 04 01:22:44.447319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.447241 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" event={"ID":"c3c7e3fb-2235-4375-9ed4-d5707052ac3b","Type":"ContainerStarted","Data":"ebfc4dfeb7640b0fea70a163445b1a610f96f9f95d7fd8896e7da5caf978c66d"} Nov 04 01:22:44.455433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.455402 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" event={"ID":"07cccfa6-abc9-44a4-9a84-762a19956113","Type":"ContainerDied","Data":"b6ef1a779da9213e22fcc5138f57ef3856d99eacbc88a6fa70ef1d7043a80887"} Nov 04 01:22:44.455691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.455674 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6ef1a779da9213e22fcc5138f57ef3856d99eacbc88a6fa70ef1d7043a80887" Nov 04 01:22:44.455901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.455889 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv" Nov 04 01:22:44.491687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.491355 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" event={"ID":"924177ce-ffbd-40a4-bb8d-0157f301d4f3","Type":"ContainerStarted","Data":"e23f651ae1fc6fb819d0667cfd4f7fbbaf7196fed0c7e2d7c7ab9fb3d0926e8e"} Nov 04 01:22:44.491687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.491437 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" event={"ID":"924177ce-ffbd-40a4-bb8d-0157f301d4f3","Type":"ContainerStarted","Data":"c2eafbf21b152f26c83729a50077a20ee1d5d5ed7cbc1efca7e092bfab1c4c4a"} Nov 04 01:22:44.553261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.553106 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" event={"ID":"054d98d6-5005-40a5-9243-8c0414ea2dbf","Type":"ContainerStarted","Data":"b78ec4be6ffade693b1927269a172fb3df52c7a11330c4a6ab51c78b3ac934f3"} Nov 04 01:22:44.594195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.586081 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" event={"ID":"cf624c2c-5046-4423-8457-bf6e62aefad5","Type":"ContainerStarted","Data":"7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36"} Nov 04 01:22:44.594195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.586165 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" event={"ID":"cf624c2c-5046-4423-8457-bf6e62aefad5","Type":"ContainerStarted","Data":"a7e5a947e052ae7ba8e6423674d76cca020a358efe1fb81895b1fb4763cf8b7f"} Nov 04 01:22:44.630873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.621975 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podStartSLOduration=2.621952566 podStartE2EDuration="2.621952566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:44.484214205 +0000 UTC m=+1503.455963077" watchObservedRunningTime="2026-11-04 01:22:44.621952566 +0000 UTC m=+1503.593701438" Nov 04 01:22:44.634161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.631481 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podStartSLOduration=2.631460481 podStartE2EDuration="2.631460481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:44.554418254 +0000 UTC m=+1503.526167126" watchObservedRunningTime="2026-11-04 01:22:44.631460481 +0000 UTC m=+1503.603209354" Nov 04 01:22:44.638715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.638676 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" event={"ID":"5541dbc1-541d-4e63-ab47-5dd90fe99545","Type":"ContainerStarted","Data":"8f90df9f1bf31845bf22fce2b65a3c43361e28757406ae4733e2d6fc7be8e7d4"} Nov 04 01:22:44.638851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.638768 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" event={"ID":"5541dbc1-541d-4e63-ab47-5dd90fe99545","Type":"ContainerStarted","Data":"1cd3de94ea8674a501c44f4446788b1507e491934e548bea926b5f5420f67fc4"} Nov 04 01:22:44.677356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.673500 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podStartSLOduration=2.673485244 podStartE2EDuration="2.673485244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:44.671012371 +0000 UTC m=+1503.642761267" watchObservedRunningTime="2026-11-04 01:22:44.673485244 +0000 UTC m=+1503.645234114" Nov 04 01:22:44.731165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:44.726885 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podStartSLOduration=2.726861818 podStartE2EDuration="2.726861818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:44.713287952 +0000 UTC m=+1503.685036824" watchObservedRunningTime="2026-11-04 01:22:44.726861818 +0000 UTC m=+1503.698610693" Nov 04 01:22:45.103598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.094176 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:45.103598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.100087 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:45.380003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.219777 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.380003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.219846 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:45.380003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.228786 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.380003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.228841 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:45.380003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.254811 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:45.380003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.302874 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.380003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.302939 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:45.387644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.387607 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:45.387902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.387872 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:45.567572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.567531 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:45.739218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.731509 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:45.963154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.951411 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d72759-d28a-4941-a1dc-69c107b65b40-kube-api-access-fbzps\" (UniqueName: \"kubernetes.io/projected/e9d72759-d28a-4941-a1dc-69c107b65b40-kube-api-access-fbzps\") pod \"e9d72759-d28a-4941-a1dc-69c107b65b40\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " Nov 04 01:22:45.963506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.963477 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-openshift-ca\") pod \"e9d72759-d28a-4941-a1dc-69c107b65b40\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " Nov 04 01:22:45.963930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:45.963905 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-xks-ca\") pod \"e9d72759-d28a-4941-a1dc-69c107b65b40\" (UID: \"e9d72759-d28a-4941-a1dc-69c107b65b40\") " Nov 04 01:22:46.001471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.001376 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-openshift-ca" pod "e9d72759-d28a-4941-a1dc-69c107b65b40" (UID: "e9d72759-d28a-4941-a1dc-69c107b65b40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:22:46.078379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.078339 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-xks-ca" pod "e9d72759-d28a-4941-a1dc-69c107b65b40" (UID: "e9d72759-d28a-4941-a1dc-69c107b65b40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:22:46.079435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.079406 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.079633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.079607 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:46.144145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.140399 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:46.144145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.140438 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d72759-d28a-4941-a1dc-69c107b65b40-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:46.174830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.174781 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.231237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.230487 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d72759-d28a-4941-a1dc-69c107b65b40-kube-api-access-fbzps" pod "e9d72759-d28a-4941-a1dc-69c107b65b40" (UID: "e9d72759-d28a-4941-a1dc-69c107b65b40"). InnerVolumeSpecName "kube-api-access-fbzps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:22:46.250162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.236271 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.250162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.236335 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:46.250162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.236786 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.250162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.236822 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:46.250162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.246714 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbzps\" (UniqueName: \"kubernetes.io/projected/e9d72759-d28a-4941-a1dc-69c107b65b40-kube-api-access-fbzps\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:22:46.288264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.250746 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:46.288264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.250806 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:46.435979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.435940 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" event={"ID":"e9d72759-d28a-4941-a1dc-69c107b65b40","Type":"ContainerDied","Data":"41173ba50a23e313b0682dccd542c9bde4e8a09c55f29a46689e7438e7055105"} Nov 04 01:22:46.457841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.457808 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41173ba50a23e313b0682dccd542c9bde4e8a09c55f29a46689e7438e7055105" Nov 04 01:22:46.466495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.466459 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" event={"ID":"77aa33d6-5854-4669-90db-6608e0e51278","Type":"ContainerStarted","Data":"2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3"} Nov 04 01:22:46.466805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.466775 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" event={"ID":"77aa33d6-5854-4669-90db-6608e0e51278","Type":"ContainerStarted","Data":"be67c33f1318bd58489d39102fb86ab86438165af591eb425729616b741086be"} Nov 04 01:22:46.468755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.437106 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt" Nov 04 01:22:46.491147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.479460 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" event={"ID":"054d98d6-5005-40a5-9243-8c0414ea2dbf","Type":"ContainerStarted","Data":"08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579"} Nov 04 01:22:46.514600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.514558 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" event={"ID":"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf","Type":"ContainerStarted","Data":"8c216ce625cc4c6293d29178a4759983155cef886d19dad479ea44318d758ef9"} Nov 04 01:22:46.532733 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.532695 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" event={"ID":"c3c7e3fb-2235-4375-9ed4-d5707052ac3b","Type":"ContainerStarted","Data":"61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d"} Nov 04 01:22:46.567321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.567208 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:46.712747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.712687 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podStartSLOduration=4.712664619 podStartE2EDuration="4.712664619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:46.493424323 +0000 UTC m=+1505.465173195" watchObservedRunningTime="2026-11-04 01:22:46.712664619 +0000 UTC m=+1505.684413515" Nov 04 01:22:46.724702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.724646 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podStartSLOduration=4.724626061 podStartE2EDuration="4.724626061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:46.613841751 +0000 UTC m=+1505.585590623" watchObservedRunningTime="2026-11-04 01:22:46.724626061 +0000 UTC m=+1505.696374933" Nov 04 01:22:46.859618 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.859513 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.880178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.859890 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:46.939054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.939005 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:46.939464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.939429 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:46.991277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:46.991213 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.031257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.030457 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.070388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.067885 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podStartSLOduration=5.067863883 podStartE2EDuration="5.067863883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:22:46.70771371 +0000 UTC m=+1505.679462582" watchObservedRunningTime="2026-11-04 01:22:47.067863883 +0000 UTC m=+1506.039612756" Nov 04 01:22:47.123455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.113048 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.123455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.113115 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:47.162740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.123846 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.162740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.123913 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:47.162740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.145246 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.175171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.173473 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.175171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.173609 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:47.177238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.177194 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:47.209293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.183649 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:22:47.286273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.219079 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.358272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.358226 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:47.358532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.298187 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.358648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.358629 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:47.358755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.351147 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.367252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.367204 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:47.389258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.370908 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:47.389258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.370967 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:47.541148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.516547 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:47.541843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.541813 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:47.542003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.541991 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:47.588781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.585267 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:47.588781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.585673 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:22:47.902145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.872374 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.902145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.872774 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:22:47.930145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.904148 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:47.930145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.904621 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:22:47.982182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.957286 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:47.982182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:47.958430 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:22:48.051152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.010439 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:48.084496 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.078328 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:22:48.084496 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.079866 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:48.085958 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.085931 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:22:48.105244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.105188 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.787447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.787396 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:48.797148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.792927 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:22:48.921022 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.919981 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.921346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.921310 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:48.921504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.920043 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.921638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.921618 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:48.921756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.689088 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:48.926145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.922643 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:48.926366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:48.926340 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:49.153205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.084217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:49.153205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.087948 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.545203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.533605 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:22:49.545203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.533684 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.545203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.534053 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:49.625147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.590154 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.665151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.661728 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:49.681384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.612112 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:22:49.808327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.803685 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.812217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.823513 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.918881 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.863679 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.596760 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.597226 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.598796 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.920854 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.638463 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.920947 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:49.922152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.921289 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:22:49.940362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:49.940267 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:50.047148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.026948 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:50.047148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.036866 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:50.047148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.038061 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.060212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.052047 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.060212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.052093 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.060212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.052107 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:50.060212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.052156 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:50.092882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.074107 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.092882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.074179 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:50.092882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.091928 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.092882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.091983 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:50.093276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.092926 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:22:50.127189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.106294 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.127189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.120606 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:50.127189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.121499 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.127189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.121539 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:50.183153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.181142 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.183153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.181210 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:50.215147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.199588 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.279739 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.279690 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:50.283210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.282806 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.283436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.283411 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:50.319203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.316967 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.319203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.317027 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:50.438802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.438762 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:50.468243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.468187 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:50.468501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.465496 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:22:50.538272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.538224 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.692230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.677955 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Nov 04 01:22:50.751145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.732810 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.935288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.923267 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:50.935288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.924145 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:50.960240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.936008 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:22:50.960240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:50.936431 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:22:51.198425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.198378 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.201228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.201190 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:51.368020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.367222 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.411563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.407793 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.411563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.407857 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:51.411563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.407942 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.411563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.407965 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:51.411563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.408877 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.411563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.408921 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:51.470645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.450434 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.470645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.450490 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:51.470645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.450573 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.470645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.450596 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:51.470645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.450675 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.470645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.450697 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:51.470645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.462073 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:51.470645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.462221 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:51.551146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.550526 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.551361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.551203 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:22:51.744144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.739217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.929608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.928229 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:51.937605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.929896 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:51.954143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:51.939589 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:52.071660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.071561 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.097977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.071983 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:52.098723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.098690 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.098918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.098893 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:52.129068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.128383 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:52.224848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.192753 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.224848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.192810 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:52.252323 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.252278 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:52.253764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.253718 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:52.477204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.405195 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.477204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.405257 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:52.477204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.405367 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.477204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.405393 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:52.477204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.405461 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.477204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.405485 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:52.477204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.405541 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:52.477204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.405561 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:52.599656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.589563 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:52.903148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:52.855090 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:53.042260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.042204 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:53.043190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.042320 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:53.045159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.045130 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:22:53.094329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.094285 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:53.094546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.094351 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:53.112139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.112091 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:53.112525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.112383 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:53.137149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.136678 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:53.140980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.137876 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:22:53.183095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.182999 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:53.183380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.183335 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:53.206284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.183766 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:53.206600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.206549 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:53.290619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.290568 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:53.292226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.291736 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:53.295416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.295384 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:53.295791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.295763 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:53.424167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.422280 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:53.424167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.422472 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:53.448700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.448603 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:53.473274 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.473217 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:53.599848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.599802 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:53.903143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:53.874329 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:22:54.103146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.102463 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:54.103146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.102770 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:54.103146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.103032 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:54.233297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.204596 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:54.293056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.292992 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:54.294853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.294802 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:54.379152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.375885 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:54.379152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.375944 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:54.379152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.376490 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:54.379152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.376551 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:54.379152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.377558 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:54.379152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.377594 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:54.394302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.394259 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:54.394610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.394574 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:54.427156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.420284 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:54.525210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.451141 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:54.595691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.525517 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:54.595994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.595960 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:54.621338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.539224 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:54.621592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.621563 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:54.799255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.783514 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:54.878689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:54.878642 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:55.083331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.083235 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:55.084643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.084494 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:55.085611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.085589 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:55.085796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.085772 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:55.110596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.110549 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:55.121823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.111763 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:55.123224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.111805 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:22:55.178807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.178772 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:55.179075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.179050 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:55.179300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.179280 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:55.179441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.179422 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:55.206265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.206217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:55.268315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.268276 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:55.268606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.268574 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:55.295617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.295575 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:55.309242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.309208 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:55.309510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.309480 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:55.317860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.317821 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:55.319086 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.319060 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:55.319298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.319275 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:55.791362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.791319 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:55.827598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.827555 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:55.883273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:55.883223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:22:56.071008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.070495 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:56.071008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.070558 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:56.085367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.085330 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:56.085595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.085570 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:56.124206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.122210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:56.124206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.122559 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:56.124206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.123624 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:56.179144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.178242 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:56.179144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.178302 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:56.179144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.178919 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:56.179144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.178959 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:56.217200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.216648 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:56.266812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.257017 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:56.266812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.257081 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:56.299146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.298205 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:56.310143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.308452 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:56.310143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.308508 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:56.323196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.320427 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:56.323196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.320485 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:56.323196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.320594 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:56.797279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.797237 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:22:56.885248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:56.885201 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:57.074213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.072638 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:57.074213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.072704 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:57.092144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.087085 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:57.092144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.087170 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:57.131437 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.131255 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:57.132953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.132911 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:57.134512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.133221 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:22:57.153247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.153036 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:57.176931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.176463 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:57.176931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.176526 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:57.177689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.177664 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:57.178008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.177986 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:57.218693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.218657 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:57.280895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.280851 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:57.290204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.281207 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:57.302882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.302846 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:57.316052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.316021 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:57.316470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.316435 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:57.337885 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.337769 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:57.340573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.340533 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:57.340778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.340757 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:57.820225 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.799917 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:57.896821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:57.896771 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.071639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.071543 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:58.071966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.071933 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:58.072437 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.072413 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:22:58.086561 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.086523 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:22:58.088172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.088147 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:58.088381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.088358 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:58.133097 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.132561 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.135147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.134159 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:22:58.158619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.157403 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.232514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.202497 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:22:58.232514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.204204 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:58.232514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.204264 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:58.232514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.214291 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:22:58.232514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.230210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.257350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.257316 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:22:58.259405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.259375 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.340259 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.313348 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.340379 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.313261 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.259617 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.341189 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.337544 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.341245 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:58.355324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.343021 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.357716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.357684 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:58.386527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.386476 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:58.497147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.482343 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.824140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.824080 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:58.936257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:58.924416 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.078621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.078529 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:22:59.078925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.078891 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:22:59.091588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.091549 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:22:59.091850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.091814 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:22:59.135367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.135297 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.136591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.135968 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.166887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.166843 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.177555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.177516 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:22:59.178161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.178105 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:22:59.178588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.178049 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:22:59.178721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.178701 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:22:59.233347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.233292 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:22:59.265145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.261226 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:22:59.265145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.261285 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:22:59.289160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.286211 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:22:59.309155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.306308 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:22:59.309155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.306379 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:22:59.322572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.322537 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:22:59.322859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.322816 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:22:59.344552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.344429 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:22:59.345557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.345514 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:22:59.387293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.387247 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 04 01:22:59.825497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.824754 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.020145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:22:59.999992 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:00.158209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.137301 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:00.158209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.140452 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.167819 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.167492 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.174938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.174904 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:00.176095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.176065 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.262987 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.263052 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.263207 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.263419 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.263455 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.263517 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.263539 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.274769 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:00.284146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.274823 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:00.309276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.309240 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:00.309528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.309496 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:00.330206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.330155 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:00.330505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.330462 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:00.349349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.349290 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:00.350366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.350323 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.732887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.723706 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:00.828911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:00.828854 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.012888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.012789 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.074565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.074520 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:01.074852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.074819 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:01.092588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.092554 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:01.092847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.092819 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:01.140303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.140254 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.174277 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.174657 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.547839 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.359230 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.349547 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.349502 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.548720 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.545241 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.360341 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.386425 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.386836 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.360027 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.549286 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.360068 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.549327 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.360304 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:01.559148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.549364 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:01.846336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.846228 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:01.989255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.989215 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:01.993144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:01.989518 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.063146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.034735 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:02.073359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.073329 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:02.073617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.073595 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:02.128202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.124724 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:02.128464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.128429 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:02.177988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.177943 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:02.178305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.178258 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:02.178862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.178838 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:02.179048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.179026 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:02.287296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.287257 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:02.287575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.287540 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:02.313304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.313268 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:02.313563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.313534 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:02.321475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.321439 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:02.321681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.321650 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:02.547434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.547387 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:02.548070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.547692 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.548737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.548442 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.549349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.549012 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.549861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.549035 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:02.554488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.549065 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:02.907636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:02.907514 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:03.069931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.069382 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:03.071138 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.070875 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:03.071138 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.070926 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:03.086873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.086536 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:03.086873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.086589 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:03.182370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.182262 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:03.182676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.182616 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:03.183838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.183813 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:03.184102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.184080 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:03.277427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.277384 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:03.277628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.277448 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:03.305617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.305580 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:03.306102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.305883 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:03.324889 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.324690 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:03.324889 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.324845 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:03.549152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.548983 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.551322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.549413 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.551322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.549731 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.551322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.550016 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:03.648092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.580699 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:03.648092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.581082 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:03.931635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:03.921973 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.069143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.069042 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:04.069143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.069103 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:04.084509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.073534 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.088694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.088651 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:04.088902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.088875 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:04.170244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.170206 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:04.171485 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.171453 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:04.182190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.178603 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:04.199709 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.199650 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:04.253539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.253503 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:04.253823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.253793 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:04.309571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.309531 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:04.309869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.309831 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:04.320446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.320416 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:04.320656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.320631 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:04.559143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.553303 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.559143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.553783 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:04.559143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.554107 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:04.559143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.554495 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.584275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.584223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:04.587207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.584574 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:04.776144 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:04.766426 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:23:04.923437 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:04.923327 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.068542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.068505 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:05.069137 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.069090 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:05.078338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.078298 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.088517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.088487 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:05.088722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.088696 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:05.183690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.183578 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:05.183958 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.183924 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:05.184688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.184659 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:05.184851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.184832 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:05.253887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.253506 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:05.253887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.253571 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:05.307669 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.307633 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:05.307962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.307915 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:05.320643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.320612 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:05.320916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.320879 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:05.556316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.556259 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.556972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.556937 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.557446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.557424 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.558254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.558230 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.591962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.590210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.591962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.590587 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.834178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.833964 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:05.925322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:05.925263 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:06.071192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.071154 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.071855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.071814 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:06.081686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.081648 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:06.087536 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.087454 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.087722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.087697 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:06.176626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.176243 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.176626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.176311 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:06.177347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.177320 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.177534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.177512 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:06.258571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.258532 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.258848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.258816 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:06.316307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.316262 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.316593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.316552 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:06.337236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.337179 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:06.337577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.337491 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:06.560510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.558407 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:06.626465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.626355 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:06.628997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.626799 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.628997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.627186 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:06.628997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.627364 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:06.628997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.627389 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:06.929317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:06.929208 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.206584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.206480 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.247613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.211771 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.247801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.247648 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:07.247801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.213055 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:07.247801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.213017 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.247801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.247745 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:07.247801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.213108 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.247801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.247788 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:07.248159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.214401 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.248159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.247825 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:07.248540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.248515 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:07.248685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.248662 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.561146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.549703 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.696271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.696216 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:07.696544 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.643211 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:07.702820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.565205 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.603264 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.707269 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.644360 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.643211 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.709335 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.709468 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.643248 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.644319 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.603314 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:07.723291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.716868 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:07.937864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.937762 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:07.983144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.970390 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:07.983144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:07.970450 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:08.104525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.104485 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.104788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.104753 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:08.139550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.139499 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.139846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.139802 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:08.200394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.200308 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.200643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.200610 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:08.201184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.201159 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.201326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.201307 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:08.272261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.272204 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:08.287969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.284205 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:08.290246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.290203 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:08.360630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.343828 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.360630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.343889 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:08.360630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.345041 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.360630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.345081 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:08.361914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.361885 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:08.362134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.362090 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:08.565272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.565225 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:08.708549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.708191 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:08.727171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.708225 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:08.734850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.708245 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:08.763180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.708263 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:08.763180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.709748 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:08.763180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.715206 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:08.941356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:08.941206 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:09.071681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.071647 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.072051 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.071984 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:09.092680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.092646 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.093007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.092965 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:09.171657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.171620 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.171930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.171895 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:09.176660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.176631 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.176863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.176841 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:09.260674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.260604 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.260886 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.260680 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:09.308300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.308249 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:09.310217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.310185 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.310420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.310397 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:09.310872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.310835 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:09.340987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.330152 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:09.340987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.330224 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:09.370169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.368323 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:09.748209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.731232 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:09.748209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.736233 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:09.748209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.736705 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:09.748209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.737093 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:09.769623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.769575 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:09.770140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.770089 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:09.850589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.849273 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:09.850589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.849338 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:09.945356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.945308 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:09.987320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.987280 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:09.987643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:09.987595 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:10.081967 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.081853 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.082273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.082218 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:10.085563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.085524 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.085761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.085739 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:10.145532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.145488 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:10.145837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.145792 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:10.191696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.190500 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.191696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.190560 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:10.191696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.191217 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.191696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.191260 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:10.254934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.254552 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.254934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.254617 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:10.309218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.305344 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.309218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.305408 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:10.334222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.330512 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:10.334222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.330575 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:10.334222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.330905 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:10.631197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.625227 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:10.734056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.732212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:10.756176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.740206 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:10.756176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.741243 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:10.756176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.741538 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:10.781136 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.772200 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:10.781136 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.772563 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:10.951611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:10.951512 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:11.110144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.091095 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.110144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.091171 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:11.110144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.091688 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.110144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.091728 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:11.170911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.170533 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.170911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.170597 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:11.236229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.226417 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.236229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.226479 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:11.255586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.255217 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.255586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.255274 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:11.349382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.349331 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:11.360242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.360196 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Nov 04 01:23:11.379270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.379232 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.379514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.379483 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:11.387145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.381603 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:11.431245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.431203 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:11.527685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.527376 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:11.740114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.738992 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:11.768653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.744202 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:11.768653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.745961 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:11.768653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.746741 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:11.786960 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.772419 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:11.786960 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.784732 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:11.972170 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:11.969272 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:12.108177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.108070 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.108443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.108399 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:12.109264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.109228 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.109427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.109407 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:12.205982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.205951 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.226383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.226333 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:12.228939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.228910 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.229252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.229226 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:12.260749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.260716 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.261044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.261014 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:12.317412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.317014 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.317412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.317077 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:12.325283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.324911 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:12.325283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.324974 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:12.383327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.383225 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:12.427334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.427263 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:12.428090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.428042 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:12.795335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.795286 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:12.795980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.795948 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:12.796233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.796195 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:12.806234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.806201 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:12.806717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.806690 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:12.822143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.815825 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:12.996156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:12.994301 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:13.091072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.090978 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:13.091362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.091301 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:13.175765 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.175710 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 04 01:23:13.176038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.175994 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 04 01:23:13.176619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.176596 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:13.176806 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.176782 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:13.266620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.266582 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:13.266899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.266868 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:13.306526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.306485 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:13.306805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.306771 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:13.318490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.318460 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:13.318725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.318698 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:13.392208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.384355 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:13.805483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.798528 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:13.805483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.798901 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:13.805483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.799178 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:13.806750 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.806706 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:13.806972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.806942 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:13.830781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.830748 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:13.994661 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:13.994582 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:14.060544 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.060451 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:23:14.156153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.148464 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:14.156700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.156666 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:14.192161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.176304 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:14.192161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.176352 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:14.192161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.176398 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:23:14.192161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.177063 5051 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d"} pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:23:14.192161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.177109 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" containerID="cri-o://61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d" gracePeriod=30 Nov 04 01:23:14.295959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.292270 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:14.295959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.292338 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:14.311334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.311250 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:14.311557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.311524 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:14.319879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.319852 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:14.320116 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.320060 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:14.388448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.388317 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:14.814143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.800202 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:14.814143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.800557 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:14.819324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.819285 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:14.819979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.819951 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:14.832644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:14.832615 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:15.004146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.003207 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:15.103193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.099409 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:23:15.116153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.114535 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:23:15.119284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.119254 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:15.119494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.119467 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:15.119657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.119642 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:23:15.120457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.120430 5051 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"8f90df9f1bf31845bf22fce2b65a3c43361e28757406ae4733e2d6fc7be8e7d4"} pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:23:15.120638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.120615 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" containerID="cri-o://8f90df9f1bf31845bf22fce2b65a3c43361e28757406ae4733e2d6fc7be8e7d4" gracePeriod=30 Nov 04 01:23:15.219260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.209318 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:15.219260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.209391 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:15.262757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.262355 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:15.262757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.262424 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:15.543215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.462995 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:15.808221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.804391 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:15.808221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.804675 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:15.808221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.804918 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:15.839143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.830161 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:23:15.839143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.831521 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:15.839143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.831843 5051 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36"} pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:23:15.839143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.831919 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" containerID="cri-o://7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36" gracePeriod=30 Nov 04 01:23:15.840484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.840452 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:15.840756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.840731 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:15.840974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:15.840953 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:16.012386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.012339 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:16.371314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.371273 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:16.371607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.371573 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:16.372332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.372307 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:16.372530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.372509 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:16.463274 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.463217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:16.815284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.815235 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:16.824383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.824341 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:16.833774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.824942 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:16.983168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.836615 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:23:16.983168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.849058 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:16.983168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.852058 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:16.983168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:16.852089 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:17.121250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.047665 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.202826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.202369 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.533211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.514444 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.550232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.533770 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:17.550232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.533852 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.550232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.533969 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:17.550232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.533998 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.760407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.760369 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:17.760604 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.760447 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.849406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.849301 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.849951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.849918 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.987314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.987262 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:17.991250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.991216 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:18.152203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:17.993182 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:18.182059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.014193 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:18.182672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.063101 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:18.183183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.063442 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:23:18.183759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.183740 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:23:18.183960 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.183932 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:23:18.185146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.185104 5051 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"8c216ce625cc4c6293d29178a4759983155cef886d19dad479ea44318d758ef9"} pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:23:18.185374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.185334 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" containerID="cri-o://8c216ce625cc4c6293d29178a4759983155cef886d19dad479ea44318d758ef9" gracePeriod=30 Nov 04 01:23:18.200146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.197429 5051 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579"} pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:23:18.200773 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.200743 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" containerID="cri-o://08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579" gracePeriod=30 Nov 04 01:23:18.208159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.202110 5051 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3"} pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" containerMessage="Container istio-proxy failed startup probe, will be restarted" Nov 04 01:23:18.208482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.208433 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" containerID="cri-o://2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3" gracePeriod=30 Nov 04 01:23:18.517947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.517717 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:18.526326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.526273 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:18.792220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.792114 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:23:18.849873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.849833 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:18.851854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:18.850330 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:19.036150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.035227 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:19.193202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.191223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:19.193202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.191637 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:19.206671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.193459 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:19.289151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.288860 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:19.382163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.366249 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 04 01:23:19.536146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.534405 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:19.852200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.851602 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:19.852385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:19.852196 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:20.091151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.089381 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:20.192415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.192320 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:20.195231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.195190 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:20.195783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.195424 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:20.196374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.195450 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:20.536240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.536198 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:20.631268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.631222 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:20.854613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.854549 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:20.855065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.854940 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:20.891153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.890502 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:23:20.948795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.943925 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:20.948795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.944038 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:20.948795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.944087 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:20.988160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:20.986979 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:23:21.287418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.287375 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:21.325754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.306452 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:21.431420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.431360 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:21.433175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.433141 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:23:21.561641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.547777 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:23:21.561641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.547878 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:23:21.561641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.559318 5051 generic.go:356] "Generic (PLEG): container finished" podID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerID="8f90df9f1bf31845bf22fce2b65a3c43361e28757406ae4733e2d6fc7be8e7d4" exitCode=0 Nov 04 01:23:21.561641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.559484 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" event={"ID":"5541dbc1-541d-4e63-ab47-5dd90fe99545","Type":"ContainerDied","Data":"8f90df9f1bf31845bf22fce2b65a3c43361e28757406ae4733e2d6fc7be8e7d4"} Nov 04 01:23:21.597975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.597469 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:23:21.640062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.640032 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:23:21.744682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.728823 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:21.744682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.728927 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:21.744682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:21.728975 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:22.602378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:22.602334 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" event={"ID":"5541dbc1-541d-4e63-ab47-5dd90fe99545","Type":"ContainerStarted","Data":"7df72b0c28079c8516b47ed377cb4382e2906345e38c8562b46e1aef22e9afdc"} Nov 04 01:23:22.606025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:22.605167 5051 generic.go:356] "Generic (PLEG): container finished" podID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerID="61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d" exitCode=0 Nov 04 01:23:22.606025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:22.605260 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" event={"ID":"c3c7e3fb-2235-4375-9ed4-d5707052ac3b","Type":"ContainerDied","Data":"61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d"} Nov 04 01:23:22.606025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:22.605290 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" event={"ID":"c3c7e3fb-2235-4375-9ed4-d5707052ac3b","Type":"ContainerStarted","Data":"7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c"} Nov 04 01:23:23.071694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.071657 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:23:23.093495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.093455 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:23.093868 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.093819 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:23.184166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.184133 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:23:23.204148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.190253 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:23.204148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.191289 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:23.892151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.892041 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:23.892635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.892176 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:23.892635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:23.892224 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:24.085257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.085222 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:24.085510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.085480 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:24.181543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.181504 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:24.181805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.181775 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:24.363407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.363367 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn"] Nov 04 01:23:24.385146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.364766 5051 cpu_manager.go:414] "containerMap: removing container" podUID="07cccfa6-abc9-44a4-9a84-762a19956113" containerName="revoke-keys" Nov 04 01:23:24.387991 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.387682 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07cccfa6-abc9-44a4-9a84-762a19956113" containerName="revoke-keys" Nov 04 01:23:24.388322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.388299 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e9d72759-d28a-4941-a1dc-69c107b65b40" containerName="revoke-keys" Nov 04 01:23:24.388424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.388412 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d72759-d28a-4941-a1dc-69c107b65b40" containerName="revoke-keys" Nov 04 01:23:24.388801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.388783 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d72759-d28a-4941-a1dc-69c107b65b40" containerName="revoke-keys" Nov 04 01:23:24.435142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.431792 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="07cccfa6-abc9-44a4-9a84-762a19956113" containerName="revoke-keys" Nov 04 01:23:24.631946 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.631785 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.638542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.638471 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:23:24.639505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.639481 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:23:24.639735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.639720 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:23:24.639924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.639909 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:23:24.640094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.640079 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn"] Nov 04 01:23:24.651004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.649394 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6"] Nov 04 01:23:24.760178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.744378 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z787d\" (UniqueName: \"kubernetes.io/projected/700b6217-0c45-4b58-9ddc-4b0ba67c5505-kube-api-access-z787d\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.760178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.744611 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.760178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.744854 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.760178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.745248 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6"] Nov 04 01:23:24.760178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.745284 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7"] Nov 04 01:23:24.760178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.746162 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:24.761214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.761189 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7"] Nov 04 01:23:24.761463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.761446 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.854956 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.855066 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.855148 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.855198 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z787d\" (UniqueName: \"kubernetes.io/projected/700b6217-0c45-4b58-9ddc-4b0ba67c5505-kube-api-access-z787d\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.855250 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjfp\" (UniqueName: \"kubernetes.io/projected/235a54ac-8745-4f3b-bf01-d14af58c95c3-kube-api-access-bjjfp\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.855328 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.855418 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.855714 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.865187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.855880 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkv5j\" (UniqueName: \"kubernetes.io/projected/ad0530ca-01be-4160-8f8f-4c7c778fe169-kube-api-access-lkv5j\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:24.909187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.881878 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.909187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.891798 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.909187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.897960 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z787d\" (UniqueName: \"kubernetes.io/projected/700b6217-0c45-4b58-9ddc-4b0ba67c5505-kube-api-access-z787d\") pod \"maas-api-revoke-sub-d0b521bb-phdbn\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:24.993159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.966941 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:24.993159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.967207 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkv5j\" (UniqueName: \"kubernetes.io/projected/ad0530ca-01be-4160-8f8f-4c7c778fe169-kube-api-access-lkv5j\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:24.993159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.967340 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:24.993159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.967390 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:24.993159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.967456 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjfp\" (UniqueName: \"kubernetes.io/projected/235a54ac-8745-4f3b-bf01-d14af58c95c3-kube-api-access-bjjfp\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:24.993159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.967524 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:24.993159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:24.990438 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:25.012025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.011994 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:25.069311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.069252 5051 generic.go:356] "Generic (PLEG): container finished" podID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerID="7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36" exitCode=0 Nov 04 01:23:25.083311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.075577 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:25.083880 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.083833 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" event={"ID":"cf624c2c-5046-4423-8457-bf6e62aefad5","Type":"ContainerDied","Data":"7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36"} Nov 04 01:23:25.084174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.084154 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" event={"ID":"cf624c2c-5046-4423-8457-bf6e62aefad5","Type":"ContainerStarted","Data":"2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5"} Nov 04 01:23:25.116345 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.095062 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:25.147140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.120621 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:25.191175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.190278 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:25.300711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.260621 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:25.300711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.260679 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:25.384041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.384005 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:25.457047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.441410 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjfp\" (UniqueName: \"kubernetes.io/projected/235a54ac-8745-4f3b-bf01-d14af58c95c3-kube-api-access-bjjfp\") pod \"maas-api-revoke-sub-e38ec51f-2jvt6\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:25.485414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.485382 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:25.534423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.527377 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkv5j\" (UniqueName: \"kubernetes.io/projected/ad0530ca-01be-4160-8f8f-4c7c778fe169-kube-api-access-lkv5j\") pod \"maas-api-revoke-sub-1c00ce92-qwgf7\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:25.722849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.722735 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:25.723172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.723140 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:25.735576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.732330 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:25.740983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:25.740946 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:26.078302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.077871 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:26.078302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.077927 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:26.128185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.128152 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:23:26.128490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.128466 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:26.128670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.128646 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:26.263505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.263469 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:26.263793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.263757 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:26.793301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.752104 5051 generic.go:356] "Generic (PLEG): container finished" podID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerID="08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579" exitCode=0 Nov 04 01:23:26.808010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.807905 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" event={"ID":"054d98d6-5005-40a5-9243-8c0414ea2dbf","Type":"ContainerDied","Data":"08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579"} Nov 04 01:23:26.893161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.880308 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:26.978306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:26.976340 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:27.090191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.077864 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:27.162206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.120205 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:27.162206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.128907 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:27.162206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.128959 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:27.162206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.150685 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:27.162206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.153668 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:27.238146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.208058 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:27.238146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.208146 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:27.246017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.242255 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:27.259262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.259217 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:27.269140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.261761 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:27.393331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.393229 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:27.675197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.626300 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:27.972459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:27.937602 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:28.035440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.031208 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:28.112158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.085264 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:28.139354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.122470 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:28.157257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.157210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:28.296210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.233099 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:28.296210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.233175 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:28.296445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.296210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:28.409145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.388177 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:28.409145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.388266 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:28.409145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.401526 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:28.439146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.438256 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:28.439146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.438319 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:28.514150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.497199 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:28.514150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.497312 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:28.514150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.497360 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:28.547612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.547514 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:23:28.653379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.652861 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:28.669735 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.484s" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.669781 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn"] Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.669869 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.669891 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6"] Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.669981 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.669998 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.670023 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.670063 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.670097 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.670152 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.670170 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.670215 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.670252 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:23:28.691145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.670288 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:23:28.820009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.806637 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" event={"ID":"235a54ac-8745-4f3b-bf01-d14af58c95c3","Type":"ContainerStarted","Data":"86e985479132251582b6948d0acac3fd8c07391a3f133cf39be857a0e52e7884"} Nov 04 01:23:28.961151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:28.939456 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.267352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.054320 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" event={"ID":"700b6217-0c45-4b58-9ddc-4b0ba67c5505","Type":"ContainerStarted","Data":"49235c9ed18498721a964d7004f595037562a3889c9cd220f863d878e108242e"} Nov 04 01:23:29.287803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.055758 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.317176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.086509 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.317176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.307687 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.427425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.325277 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:29.467158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.371193 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:23:29.475146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.372912 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.526200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.427668 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:29.585146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.444210 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:29.819201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:29.673534 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:30.481146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.197937 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:30.481146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.460488 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:30.481146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.324273 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:30.481146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.477269 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:30.498475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.494421 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:30.498475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.495650 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:30.527208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.500869 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:30.747196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.621731 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:23:30.624591 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0530ca_01be_4160_8f8f_4c7c778fe169.slice/crio-978ec65bf3748ffd38f04a80680b8a6344271800dfb9f697cc19a2227b497648 WatchSource:0}: Error finding container 978ec65bf3748ffd38f04a80680b8a6344271800dfb9f697cc19a2227b497648: Status 404 returned error can't find the container with id 978ec65bf3748ffd38f04a80680b8a6344271800dfb9f697cc19a2227b497648 Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.625248 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.838835 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.023624 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.869405 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.023701 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.907451 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.023761 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.327303 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:31.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.023809 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.237584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.237542 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:31.237888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.237842 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:31.362021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:30.847066 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:31.362021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.247871 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:31.362021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.247998 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:23:31.395541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.395507 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:31.395842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.395808 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:31.420514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.413367 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:31.420514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.413440 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:31.509326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.506776 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.509326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.507931 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.509326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.508580 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.509326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.509268 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.561015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.479718 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:31.561372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.561316 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:31.809610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.620309 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Nov 04 01:23:31.809856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.620357 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:31.810070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.635269 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:31.810242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.810212 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:31.810350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.752589 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.810456 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.683104 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.820257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.820208 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:31.952895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:31.940414 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:32.057202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.014962 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:32.141145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.070777 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:32.392150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.382679 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:32.416454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.416415 5051 generic.go:356] "Generic (PLEG): container finished" podID="77aa33d6-5854-4669-90db-6608e0e51278" containerID="2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3" exitCode=0 Nov 04 01:23:32.490881 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:32.490847 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.265s" Nov 04 01:23:32.491226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.491108 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" event={"ID":"77aa33d6-5854-4669-90db-6608e0e51278","Type":"ContainerDied","Data":"2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3"} Nov 04 01:23:32.549572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.543449 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:32.549572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.543512 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:32.549572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.543604 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:32.549572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.543628 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:32.566162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.552068 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:32.566162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.552113 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:32.618205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.611537 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:32.618205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.611598 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:32.665843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.665792 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:32.666506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.666473 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:32.667013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.666991 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:32.689429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.667418 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:32.904717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.888264 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:32.913260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:32.913197 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:33.079940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.079889 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:33.080251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.080215 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:33.300063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.293053 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:33.380682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.380583 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:33.380980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.297229 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:33.399430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.399386 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:33.532301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.516391 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:33.532301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.518045 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:33.532301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.518408 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:33.554656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.554564 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:33.686247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.686192 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:33.693203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.686880 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:33.768599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.693640 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:33.768599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:33.701220 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:34.162174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.095588 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:34.165755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.165718 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:34.170141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.111881 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:34.191274 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.191231 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:34.192211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.192178 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" event={"ID":"77aa33d6-5854-4669-90db-6608e0e51278","Type":"ContainerStarted","Data":"13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c"} Nov 04 01:23:34.294402 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.291455 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:34.294402 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.291508 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:34.361794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.334407 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:23:34.369292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.369244 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:34.369562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.369513 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:34.371751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.371608 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" event={"ID":"054d98d6-5005-40a5-9243-8c0414ea2dbf","Type":"ContainerStarted","Data":"33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a"} Nov 04 01:23:34.488740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.488709 5051 generic.go:356] "Generic (PLEG): container finished" podID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerID="8c216ce625cc4c6293d29178a4759983155cef886d19dad479ea44318d758ef9" exitCode=0 Nov 04 01:23:34.511465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.493173 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" event={"ID":"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf","Type":"ContainerDied","Data":"8c216ce625cc4c6293d29178a4759983155cef886d19dad479ea44318d758ef9"} Nov 04 01:23:34.519260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.519215 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:34.527632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.519507 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:34.528610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.519532 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:34.531063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.531033 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" event={"ID":"ad0530ca-01be-4160-8f8f-4c7c778fe169","Type":"ContainerStarted","Data":"978ec65bf3748ffd38f04a80680b8a6344271800dfb9f697cc19a2227b497648"} Nov 04 01:23:34.601367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.601326 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:34.629476 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.629388 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7"] Nov 04 01:23:34.742658 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.741744 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:34.769615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.768576 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:34.769615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.768896 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:34.788919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.788884 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:34.821835 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:34.793140 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:23:34.968684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:34.968541 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" podStartSLOduration=10.96850464 podStartE2EDuration="10.96850464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:34.44260185 +0000 UTC m=+1553.414350722" watchObservedRunningTime="2026-11-04 01:23:34.96850464 +0000 UTC m=+1553.940253512" Nov 04 01:23:35.075148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.071534 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:35.075148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.071610 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:35.100145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.092566 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:35.100145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.092684 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:35.179297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.178329 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:35.179297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.178405 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:35.308399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.308350 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:35.308716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.308663 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:35.354159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.353105 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:35.354159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.353207 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:35.612132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.544315 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:35.612132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.555360 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:35.612132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.555721 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:35.670267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.641043 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:35.670267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.643417 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:23:35.693247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.684778 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:23:35.753856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.753811 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:35.783214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.780259 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:35.783214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.780209 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:35.859193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:35.854832 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:36.037214 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:36.037163 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0530ca_01be_4160_8f8f_4c7c778fe169.slice/crio-conmon-54fc9fcd0554d979139a38f7b620d2b86189dcb9c84734b0b2b9b4baa8b96b8e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0530ca_01be_4160_8f8f_4c7c778fe169.slice/crio-54fc9fcd0554d979139a38f7b620d2b86189dcb9c84734b0b2b9b4baa8b96b8e.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:23:36.037885 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:36.037646 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0530ca_01be_4160_8f8f_4c7c778fe169.slice/crio-conmon-54fc9fcd0554d979139a38f7b620d2b86189dcb9c84734b0b2b9b4baa8b96b8e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0530ca_01be_4160_8f8f_4c7c778fe169.slice/crio-54fc9fcd0554d979139a38f7b620d2b86189dcb9c84734b0b2b9b4baa8b96b8e.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:23:36.103444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.103404 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:36.103689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.103660 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:36.108365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.108332 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:36.108576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.108551 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:36.239065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.239028 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:36.281649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.281610 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:36.319197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.316918 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:36.319197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.316980 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:36.340531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.336980 5051 generic.go:356] "Generic (PLEG): container finished" podID="235a54ac-8745-4f3b-bf01-d14af58c95c3" containerID="77f66924ba5a83b1d068ee39b6b07eafca13044d7906195e5e28cdc4f4e5117a" exitCode=0 Nov 04 01:23:36.340531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.337163 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" event={"ID":"235a54ac-8745-4f3b-bf01-d14af58c95c3","Type":"ContainerDied","Data":"77f66924ba5a83b1d068ee39b6b07eafca13044d7906195e5e28cdc4f4e5117a"} Nov 04 01:23:36.357534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.357495 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:36.357857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.357825 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:36.545840 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.545794 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:36.663237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.663090 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:36.664004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.548866 5051 generic.go:356] "Generic (PLEG): container finished" podID="700b6217-0c45-4b58-9ddc-4b0ba67c5505" containerID="2d77a33b045253fad50f7b4053124851abd2bbb12bb7025e159aa6648f0d6f91" exitCode=0 Nov 04 01:23:36.665572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.548895 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" event={"ID":"700b6217-0c45-4b58-9ddc-4b0ba67c5505","Type":"ContainerDied","Data":"2d77a33b045253fad50f7b4053124851abd2bbb12bb7025e159aa6648f0d6f91"} Nov 04 01:23:36.666184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.565958 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:36.666488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.565992 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:36.691273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.691228 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:36.810556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.805437 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:36.810556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.808267 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:23:36.865166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.855744 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:36.867085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:36.867044 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" event={"ID":"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf","Type":"ContainerStarted","Data":"58e152501bdacba54c6f9683cf4a6572a98cd4a2a4fedf9b345559c786a74387"} Nov 04 01:23:37.137342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.137306 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:23:37.137591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.137566 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" event={"ID":"ad0530ca-01be-4160-8f8f-4c7c778fe169","Type":"ContainerStarted","Data":"54fc9fcd0554d979139a38f7b620d2b86189dcb9c84734b0b2b9b4baa8b96b8e"} Nov 04 01:23:37.138666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.107972 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:37.138852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.138829 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:37.139238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.124213 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 04 01:23:37.139381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.139360 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 04 01:23:37.196764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.196733 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:37.213368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.213329 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:37.241259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.241211 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:37.355164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.346553 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:37.355164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.346627 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:37.355164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.346994 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:37.355164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.347023 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:37.360218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.360172 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:37.396310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.360446 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:37.470584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.470550 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:23:37.684204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.682166 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:37.684204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.682560 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:37.689153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.685368 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:37.809622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.809557 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:23:37.810150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.810039 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:37.862154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:37.856949 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:38.010266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.010229 5051 generic.go:356] "Generic (PLEG): container finished" podID="ad0530ca-01be-4160-8f8f-4c7c778fe169" containerID="54fc9fcd0554d979139a38f7b620d2b86189dcb9c84734b0b2b9b4baa8b96b8e" exitCode=0 Nov 04 01:23:38.011173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.011140 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" event={"ID":"ad0530ca-01be-4160-8f8f-4c7c778fe169","Type":"ContainerDied","Data":"54fc9fcd0554d979139a38f7b620d2b86189dcb9c84734b0b2b9b4baa8b96b8e"} Nov 04 01:23:38.092144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.091658 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:38.092144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.091730 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:38.237193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.193520 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 04 01:23:38.237193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.193583 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 04 01:23:38.476213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.450607 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:38.476213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.475433 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:23:38.478099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.477829 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:38.478099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.477892 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:38.478099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.477910 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:38.478099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.477958 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:38.519275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.519229 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:38.523475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.523426 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:38.722397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.722311 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:38.722917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.722754 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:38.723114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.723085 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:38.829149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.813382 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:38.854145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.826515 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:38.854145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.837861 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:38.854145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.850330 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:23:38.872146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:38.866916 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:39.197413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.197215 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:39.197413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.197285 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:39.400151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.374194 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:39.431250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.431200 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:39.431544 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.374225 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:39.432078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.432045 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:39.432538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.400712 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:39.432695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.432672 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:39.473224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.450754 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:39.500142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.453209 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:23:39.500447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.453284 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:39.543150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.539599 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:39.543150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.539662 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:39.764304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.735539 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:39.764304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.735867 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:39.764304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.736058 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:39.914977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.914933 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:39.982208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.945200 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:39.982208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.971780 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:39.982208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:39.972689 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:40.179367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.179329 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:40.179659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.179614 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:40.219296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.219254 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:40.219595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.219565 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:40.490658 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.490620 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:40.490898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.490872 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:40.691284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.691221 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:40.744426 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.741663 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:40.910186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.741799 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:40.910186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.742892 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:40.910186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.838638 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:40.910186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.744640 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:40.910186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.742935 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:40.910186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.839068 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:40.910186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.868029 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:40.993684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.967761 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:40.993684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.968140 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:41.019172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.994205 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:23:41.019975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:40.994270 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:41.161147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.107566 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:41.161147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.107630 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:41.195089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.169341 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:41.475458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.471321 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:41.678553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.678446 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:41.681924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.681885 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:41.682161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.682137 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:41.682633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.682612 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:41.682796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.682777 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:41.706660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.706631 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:41.706874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.706852 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:41.771147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.745661 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:23:41.835292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.793600 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-openshift-ca\") pod \"235a54ac-8745-4f3b-bf01-d14af58c95c3\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " Nov 04 01:23:41.835292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.793791 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-xks-ca\") pod \"235a54ac-8745-4f3b-bf01-d14af58c95c3\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " Nov 04 01:23:41.835292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.793961 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-xks-ca\") pod \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " Nov 04 01:23:41.835292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.794000 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/700b6217-0c45-4b58-9ddc-4b0ba67c5505-kube-api-access-z787d\" (UniqueName: \"kubernetes.io/projected/700b6217-0c45-4b58-9ddc-4b0ba67c5505-kube-api-access-z787d\") pod \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " Nov 04 01:23:41.835292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.794042 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/235a54ac-8745-4f3b-bf01-d14af58c95c3-kube-api-access-bjjfp\" (UniqueName: \"kubernetes.io/projected/235a54ac-8745-4f3b-bf01-d14af58c95c3-kube-api-access-bjjfp\") pod \"235a54ac-8745-4f3b-bf01-d14af58c95c3\" (UID: \"235a54ac-8745-4f3b-bf01-d14af58c95c3\") " Nov 04 01:23:41.835292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.794079 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-openshift-ca\") pod \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\" (UID: \"700b6217-0c45-4b58-9ddc-4b0ba67c5505\") " Nov 04 01:23:41.923592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:41.923158 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-openshift-ca" pod "235a54ac-8745-4f3b-bf01-d14af58c95c3" (UID: "235a54ac-8745-4f3b-bf01-d14af58c95c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:42.154201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.143979 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:42.218248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.217050 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-xks-ca" pod "235a54ac-8745-4f3b-bf01-d14af58c95c3" (UID: "235a54ac-8745-4f3b-bf01-d14af58c95c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:42.298902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.272953 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:42.299280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.299254 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:42.390220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.309975 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 04 01:23:42.700928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.700885 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 04 01:23:42.705199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.590086 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/235a54ac-8745-4f3b-bf01-d14af58c95c3-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:42.705334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.654179 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:42.735147 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:42.319028 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.136s" Nov 04 01:23:42.799182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.737533 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:42.946329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.654574 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:42.949427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.655770 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:42.959280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.703316 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-xks-ca" pod "700b6217-0c45-4b58-9ddc-4b0ba67c5505" (UID: "700b6217-0c45-4b58-9ddc-4b0ba67c5505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:42.974155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.654553 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:42.977157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.654215 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:23:42.997344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.997306 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/700b6217-0c45-4b58-9ddc-4b0ba67c5505-kube-api-access-z787d" pod "700b6217-0c45-4b58-9ddc-4b0ba67c5505" (UID: "700b6217-0c45-4b58-9ddc-4b0ba67c5505"). InnerVolumeSpecName "kube-api-access-z787d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:43.041433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.736703 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235a54ac-8745-4f3b-bf01-d14af58c95c3-kube-api-access-bjjfp" pod "235a54ac-8745-4f3b-bf01-d14af58c95c3" (UID: "235a54ac-8745-4f3b-bf01-d14af58c95c3"). InnerVolumeSpecName "kube-api-access-bjjfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:43.137779 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.593023 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:43.142143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.138094 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:43.142409 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.630550 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-openshift-ca" pod "700b6217-0c45-4b58-9ddc-4b0ba67c5505" (UID: "700b6217-0c45-4b58-9ddc-4b0ba67c5505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:43.166211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.631011 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:43.166811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.166760 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:43.167024 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.554250 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:43.202776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.127727 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:23:43.207612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.207574 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:23:43.290147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.167491 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:43.299683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:42.753711 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:43.378183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.361762 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:43.378183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.367214 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:43.378183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.367264 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:43.399248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.394745 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:43.399248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.394804 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:43.496143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.471564 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:43.498539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.493782 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:43.795146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.769380 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:43.795460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.613478 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:43.810148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.795596 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z787d\" (UniqueName: \"kubernetes.io/projected/700b6217-0c45-4b58-9ddc-4b0ba67c5505-kube-api-access-z787d\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:43.835141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.822817 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjjfp\" (UniqueName: \"kubernetes.io/projected/235a54ac-8745-4f3b-bf01-d14af58c95c3-kube-api-access-bjjfp\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:43.835141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.822912 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/700b6217-0c45-4b58-9ddc-4b0ba67c5505-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:43.936664 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.936612 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:23:43.987146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:43.985565 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:23:44.011459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.002053 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:23:44.011459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.002614 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-openshift-ca\") pod \"ad0530ca-01be-4160-8f8f-4c7c778fe169\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " Nov 04 01:23:44.011459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.002750 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-xks-ca\") pod \"ad0530ca-01be-4160-8f8f-4c7c778fe169\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " Nov 04 01:23:44.011459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.002907 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0530ca-01be-4160-8f8f-4c7c778fe169-kube-api-access-lkv5j\" (UniqueName: \"kubernetes.io/projected/ad0530ca-01be-4160-8f8f-4c7c778fe169-kube-api-access-lkv5j\") pod \"ad0530ca-01be-4160-8f8f-4c7c778fe169\" (UID: \"ad0530ca-01be-4160-8f8f-4c7c778fe169\") " Nov 04 01:23:44.011459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.004477 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:23:44.066020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.037237 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:23:44.215512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.215468 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-xks-ca" pod "ad0530ca-01be-4160-8f8f-4c7c778fe169" (UID: "ad0530ca-01be-4160-8f8f-4c7c778fe169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:44.232305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.227971 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:44.245144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.242549 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-openshift-ca" pod "ad0530ca-01be-4160-8f8f-4c7c778fe169" (UID: "ad0530ca-01be-4160-8f8f-4c7c778fe169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:44.285147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.260606 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0530ca-01be-4160-8f8f-4c7c778fe169-kube-api-access-lkv5j" pod "ad0530ca-01be-4160-8f8f-4c7c778fe169" (UID: "ad0530ca-01be-4160-8f8f-4c7c778fe169"). InnerVolumeSpecName "kube-api-access-lkv5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:44.334210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.281299 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:23:44.334210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.332886 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:44.334985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.334773 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:44.335402 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.335366 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:44.396143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.392080 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:44.445146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.407008 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:44.445146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.392274 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:44.445146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.408249 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:44.445146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.420308 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:44.523902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.507870 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:44.680213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.674198 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" Nov 04 01:23:44.841548 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:44.841503 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.612s" Nov 04 01:23:44.841853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.841839 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:23:44.842042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.842017 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:23:44.842399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.842387 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:23:44.842543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.842533 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:23:44.846503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.846483 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908189 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908250 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" event={"ID":"700b6217-0c45-4b58-9ddc-4b0ba67c5505","Type":"ContainerDied","Data":"49235c9ed18498721a964d7004f595037562a3889c9cd220f863d878e108242e"} Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908289 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49235c9ed18498721a964d7004f595037562a3889c9cd220f863d878e108242e" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908305 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6" event={"ID":"235a54ac-8745-4f3b-bf01-d14af58c95c3","Type":"ContainerDied","Data":"86e985479132251582b6948d0acac3fd8c07391a3f133cf39be857a0e52e7884"} Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908328 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86e985479132251582b6948d0acac3fd8c07391a3f133cf39be857a0e52e7884" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908361 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908388 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908443 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908480 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908497 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908527 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.908557 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.685752 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkv5j\" (UniqueName: \"kubernetes.io/projected/ad0530ca-01be-4160-8f8f-4c7c778fe169-kube-api-access-lkv5j\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:44.912177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.886026 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:44.954518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.954420 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" event={"ID":"ad0530ca-01be-4160-8f8f-4c7c778fe169","Type":"ContainerDied","Data":"978ec65bf3748ffd38f04a80680b8a6344271800dfb9f697cc19a2227b497648"} Nov 04 01:23:44.954740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.954722 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="978ec65bf3748ffd38f04a80680b8a6344271800dfb9f697cc19a2227b497648" Nov 04 01:23:44.957865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.957840 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad0530ca-01be-4160-8f8f-4c7c778fe169-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:44.969415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:44.969386 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7" Nov 04 01:23:45.023785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.019187 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn" Nov 04 01:23:45.030270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.030231 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:45.047289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.032956 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:23:45.047289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.033029 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:23:45.047289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.039709 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:45.201241 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:45.197236 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235a54ac_8745_4f3b_bf01_d14af58c95c3.slice/crio-86e985479132251582b6948d0acac3fd8c07391a3f133cf39be857a0e52e7884\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700b6217_0c45_4b58_9ddc_4b0ba67c5505.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700b6217_0c45_4b58_9ddc_4b0ba67c5505.slice/crio-49235c9ed18498721a964d7004f595037562a3889c9cd220f863d878e108242e\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235a54ac_8745_4f3b_bf01_d14af58c95c3.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:23:45.311315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.197752 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:23:45.311315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.273461 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:45.311315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.285472 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:45.311315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.310836 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:45.311315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.310887 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:45.369672 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:23:45.369620 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235a54ac_8745_4f3b_bf01_d14af58c95c3.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:23:45.553397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.553354 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 04 01:23:45.553794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.553737 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 04 01:23:45.659445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.659400 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:23:45.798310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.797056 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:23:45.799054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.799033 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:23:45.839582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:45.839500 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:23:46.275144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.256504 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:46.275144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.256583 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:46.305977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.305609 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 04 01:23:46.305977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.305675 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 04 01:23:46.973224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.973184 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7"] Nov 04 01:23:46.974406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974376 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ad0530ca-01be-4160-8f8f-4c7c778fe169" containerName="revoke-keys" Nov 04 01:23:46.974406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974405 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0530ca-01be-4160-8f8f-4c7c778fe169" containerName="revoke-keys" Nov 04 01:23:46.974625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974427 5051 cpu_manager.go:414] "containerMap: removing container" podUID="235a54ac-8745-4f3b-bf01-d14af58c95c3" containerName="revoke-keys" Nov 04 01:23:46.974625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974435 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="235a54ac-8745-4f3b-bf01-d14af58c95c3" containerName="revoke-keys" Nov 04 01:23:46.974625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974468 5051 cpu_manager.go:414] "containerMap: removing container" podUID="700b6217-0c45-4b58-9ddc-4b0ba67c5505" containerName="revoke-keys" Nov 04 01:23:46.974625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974476 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="700b6217-0c45-4b58-9ddc-4b0ba67c5505" containerName="revoke-keys" Nov 04 01:23:46.974825 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974653 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="700b6217-0c45-4b58-9ddc-4b0ba67c5505" containerName="revoke-keys" Nov 04 01:23:46.974825 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974682 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0530ca-01be-4160-8f8f-4c7c778fe169" containerName="revoke-keys" Nov 04 01:23:46.974825 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.974696 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="235a54ac-8745-4f3b-bf01-d14af58c95c3" containerName="revoke-keys" Nov 04 01:23:46.988146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:46.983420 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.002153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.000516 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:23:47.002153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.000658 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:23:47.002489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.002464 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:23:47.006144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.002656 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:23:47.037151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.014229 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7"] Nov 04 01:23:47.115326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.080811 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.115326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.080866 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.115326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.081057 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qw6\" (UniqueName: \"kubernetes.io/projected/2f27d705-1a3c-44d3-99b2-126371818776-kube-api-access-78qw6\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.196149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.185490 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:23:47.196149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.188418 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.196149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.188467 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.196149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.188593 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qw6\" (UniqueName: \"kubernetes.io/projected/2f27d705-1a3c-44d3-99b2-126371818776-kube-api-access-78qw6\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.196149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.193920 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:23:47.243191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.233033 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qw6\" (UniqueName: \"kubernetes.io/projected/2f27d705-1a3c-44d3-99b2-126371818776-kube-api-access-78qw6\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.266014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.262186 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Nov 04 01:23:47.266014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.262244 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Nov 04 01:23:47.266014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.264477 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.281215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.278573 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e2fb6d-lzzd7\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.384646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.384048 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:47.719072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.719032 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn"] Nov 04 01:23:47.747229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.747196 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn"] Nov 04 01:23:47.747472 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.747317 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:47.813525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.813489 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:47.813652 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.813561 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:47.813719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.813698 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb6vd\" (UniqueName: \"kubernetes.io/projected/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-kube-api-access-qb6vd\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:47.835252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.835156 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7"] Nov 04 01:23:47.948152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.916350 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:47.961572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.961538 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:47.970152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.962010 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb6vd\" (UniqueName: \"kubernetes.io/projected/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-kube-api-access-qb6vd\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:48.008412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:47.971692 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:48.033354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.025273 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb6vd\" (UniqueName: \"kubernetes.io/projected/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-kube-api-access-qb6vd\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:48.064179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.061886 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e5e5c5-j5hkn\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:48.130335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.130251 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" event={"ID":"2f27d705-1a3c-44d3-99b2-126371818776","Type":"ContainerStarted","Data":"0f160463b1e25d529897d967e50069a8dff7d846e67e6122981ecdf1b7bfd7d8"} Nov 04 01:23:48.153648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.152147 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:23:48.155098 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.155070 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:23:48.282631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.282507 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:23:48.286463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.286437 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:23:48.332568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.332419 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:48.357447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.355258 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr"] Nov 04 01:23:48.382171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.378097 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr"] Nov 04 01:23:48.382171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.378225 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.435197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.435161 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.435381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.435275 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2km5z\" (UniqueName: \"kubernetes.io/projected/f139661f-18a5-432b-8b79-cfb1302f37d6-kube-api-access-2km5z\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.435702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.435671 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.537171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.537115 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.537445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.537426 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2km5z\" (UniqueName: \"kubernetes.io/projected/f139661f-18a5-432b-8b79-cfb1302f37d6-kube-api-access-2km5z\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.537703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.537677 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.538425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.537842 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.547740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.542695 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.548040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.547973 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2km5z\" (UniqueName: \"kubernetes.io/projected/f139661f-18a5-432b-8b79-cfb1302f37d6-kube-api-access-2km5z\") pod \"maas-api-revoke-sub-e77e7540-28vmr\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.742179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.712407 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:48.851146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:48.851104 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn"] Nov 04 01:23:49.132373 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:23:49.132341 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf139661f_18a5_432b_8b79_cfb1302f37d6.slice/crio-fcb5a737b38372b4eec39b1da59f1cd565cb72baa54ac9e85056fd8b7f5c90e1 WatchSource:0}: Error finding container fcb5a737b38372b4eec39b1da59f1cd565cb72baa54ac9e85056fd8b7f5c90e1: Status 404 returned error can't find the container with id fcb5a737b38372b4eec39b1da59f1cd565cb72baa54ac9e85056fd8b7f5c90e1 Nov 04 01:23:49.133592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:49.133561 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr"] Nov 04 01:23:49.142763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:49.142733 5051 generic.go:356] "Generic (PLEG): container finished" podID="2f27d705-1a3c-44d3-99b2-126371818776" containerID="b118be58540fa403863e341d2512090d0cdcc0d0326635962eb731c9c8ed4074" exitCode=0 Nov 04 01:23:49.142884 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:49.142810 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" event={"ID":"2f27d705-1a3c-44d3-99b2-126371818776","Type":"ContainerDied","Data":"b118be58540fa403863e341d2512090d0cdcc0d0326635962eb731c9c8ed4074"} Nov 04 01:23:49.144394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:49.144371 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" event={"ID":"fe36d7d9-13ec-4ec7-b015-67f700ecb66f","Type":"ContainerStarted","Data":"1933fbbdc4992ecd838ee80fc7e384fb46974158f44561f7b9d7e2e864421c73"} Nov 04 01:23:50.157687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.157654 5051 generic.go:356] "Generic (PLEG): container finished" podID="fe36d7d9-13ec-4ec7-b015-67f700ecb66f" containerID="ecccf78e3847d6de5951e2de850c918e4e412ba961132a9fe3219ead16a0c260" exitCode=0 Nov 04 01:23:50.158090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.157780 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" event={"ID":"fe36d7d9-13ec-4ec7-b015-67f700ecb66f","Type":"ContainerDied","Data":"ecccf78e3847d6de5951e2de850c918e4e412ba961132a9fe3219ead16a0c260"} Nov 04 01:23:50.159943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.159914 5051 generic.go:356] "Generic (PLEG): container finished" podID="f139661f-18a5-432b-8b79-cfb1302f37d6" containerID="d9ab30fb474faf71d233bf3cec9d7623d95a918e9c4116225fff066ea66255db" exitCode=0 Nov 04 01:23:50.160083 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.159991 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" event={"ID":"f139661f-18a5-432b-8b79-cfb1302f37d6","Type":"ContainerDied","Data":"d9ab30fb474faf71d233bf3cec9d7623d95a918e9c4116225fff066ea66255db"} Nov 04 01:23:50.160083 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.160034 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" event={"ID":"f139661f-18a5-432b-8b79-cfb1302f37d6","Type":"ContainerStarted","Data":"fcb5a737b38372b4eec39b1da59f1cd565cb72baa54ac9e85056fd8b7f5c90e1"} Nov 04 01:23:50.448135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.448086 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:50.535521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.535489 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f27d705-1a3c-44d3-99b2-126371818776-kube-api-access-78qw6\" (UniqueName: \"kubernetes.io/projected/2f27d705-1a3c-44d3-99b2-126371818776-kube-api-access-78qw6\") pod \"2f27d705-1a3c-44d3-99b2-126371818776\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " Nov 04 01:23:50.535877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.535856 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-openshift-ca\") pod \"2f27d705-1a3c-44d3-99b2-126371818776\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " Nov 04 01:23:50.536354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.536318 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-openshift-ca" pod "2f27d705-1a3c-44d3-99b2-126371818776" (UID: "2f27d705-1a3c-44d3-99b2-126371818776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:50.536613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.536586 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:50.538545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.538493 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f27d705-1a3c-44d3-99b2-126371818776-kube-api-access-78qw6" pod "2f27d705-1a3c-44d3-99b2-126371818776" (UID: "2f27d705-1a3c-44d3-99b2-126371818776"). InnerVolumeSpecName "kube-api-access-78qw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:50.637563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.637485 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-xks-ca\") pod \"2f27d705-1a3c-44d3-99b2-126371818776\" (UID: \"2f27d705-1a3c-44d3-99b2-126371818776\") " Nov 04 01:23:50.638042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.638025 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78qw6\" (UniqueName: \"kubernetes.io/projected/2f27d705-1a3c-44d3-99b2-126371818776-kube-api-access-78qw6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:50.639651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.639624 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-xks-ca" pod "2f27d705-1a3c-44d3-99b2-126371818776" (UID: "2f27d705-1a3c-44d3-99b2-126371818776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:50.739273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:50.739242 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f27d705-1a3c-44d3-99b2-126371818776-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:51.171415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.171383 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" Nov 04 01:23:51.171865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.171419 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7" event={"ID":"2f27d705-1a3c-44d3-99b2-126371818776","Type":"ContainerDied","Data":"0f160463b1e25d529897d967e50069a8dff7d846e67e6122981ecdf1b7bfd7d8"} Nov 04 01:23:51.171865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.171498 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f160463b1e25d529897d967e50069a8dff7d846e67e6122981ecdf1b7bfd7d8" Nov 04 01:23:51.527251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.527219 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:51.530676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.530654 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:51.597980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.597941 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-kube-api-access-qb6vd\" (UniqueName: \"kubernetes.io/projected/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-kube-api-access-qb6vd\") pod \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " Nov 04 01:23:51.598239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.598217 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f139661f-18a5-432b-8b79-cfb1302f37d6-kube-api-access-2km5z\" (UniqueName: \"kubernetes.io/projected/f139661f-18a5-432b-8b79-cfb1302f37d6-kube-api-access-2km5z\") pod \"f139661f-18a5-432b-8b79-cfb1302f37d6\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " Nov 04 01:23:51.598334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.598315 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-openshift-ca\") pod \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " Nov 04 01:23:51.598453 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.598410 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-xks-ca\") pod \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\" (UID: \"fe36d7d9-13ec-4ec7-b015-67f700ecb66f\") " Nov 04 01:23:51.598534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.598458 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-xks-ca\") pod \"f139661f-18a5-432b-8b79-cfb1302f37d6\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " Nov 04 01:23:51.598803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.598640 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-openshift-ca\") pod \"f139661f-18a5-432b-8b79-cfb1302f37d6\" (UID: \"f139661f-18a5-432b-8b79-cfb1302f37d6\") " Nov 04 01:23:51.598803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.598789 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-openshift-ca" pod "fe36d7d9-13ec-4ec7-b015-67f700ecb66f" (UID: "fe36d7d9-13ec-4ec7-b015-67f700ecb66f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:51.599381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.599341 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-openshift-ca" pod "f139661f-18a5-432b-8b79-cfb1302f37d6" (UID: "f139661f-18a5-432b-8b79-cfb1302f37d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:51.599813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.599785 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:51.599902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.599818 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:51.602297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.602268 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f139661f-18a5-432b-8b79-cfb1302f37d6-kube-api-access-2km5z" pod "f139661f-18a5-432b-8b79-cfb1302f37d6" (UID: "f139661f-18a5-432b-8b79-cfb1302f37d6"). InnerVolumeSpecName "kube-api-access-2km5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:51.602385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.602318 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-kube-api-access-qb6vd" pod "fe36d7d9-13ec-4ec7-b015-67f700ecb66f" (UID: "fe36d7d9-13ec-4ec7-b015-67f700ecb66f"). InnerVolumeSpecName "kube-api-access-qb6vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:51.602908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.602883 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-xks-ca" pod "fe36d7d9-13ec-4ec7-b015-67f700ecb66f" (UID: "fe36d7d9-13ec-4ec7-b015-67f700ecb66f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:51.603062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.603037 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-xks-ca" pod "f139661f-18a5-432b-8b79-cfb1302f37d6" (UID: "f139661f-18a5-432b-8b79-cfb1302f37d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:51.701842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.701776 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:51.701842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.701811 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f139661f-18a5-432b-8b79-cfb1302f37d6-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:51.701842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.701827 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb6vd\" (UniqueName: \"kubernetes.io/projected/fe36d7d9-13ec-4ec7-b015-67f700ecb66f-kube-api-access-qb6vd\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:51.701842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:51.701843 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2km5z\" (UniqueName: \"kubernetes.io/projected/f139661f-18a5-432b-8b79-cfb1302f37d6-kube-api-access-2km5z\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:52.249737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:52.249695 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" event={"ID":"fe36d7d9-13ec-4ec7-b015-67f700ecb66f","Type":"ContainerDied","Data":"1933fbbdc4992ecd838ee80fc7e384fb46974158f44561f7b9d7e2e864421c73"} Nov 04 01:23:52.250249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:52.249752 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1933fbbdc4992ecd838ee80fc7e384fb46974158f44561f7b9d7e2e864421c73" Nov 04 01:23:52.250249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:52.249827 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn" Nov 04 01:23:52.254678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:52.254640 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" event={"ID":"f139661f-18a5-432b-8b79-cfb1302f37d6","Type":"ContainerDied","Data":"fcb5a737b38372b4eec39b1da59f1cd565cb72baa54ac9e85056fd8b7f5c90e1"} Nov 04 01:23:52.254838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:52.254697 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcb5a737b38372b4eec39b1da59f1cd565cb72baa54ac9e85056fd8b7f5c90e1" Nov 04 01:23:52.254838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:52.254764 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr" Nov 04 01:23:54.577971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.577933 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g"] Nov 04 01:23:54.578647 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578631 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fe36d7d9-13ec-4ec7-b015-67f700ecb66f" containerName="revoke-keys" Nov 04 01:23:54.578701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578647 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe36d7d9-13ec-4ec7-b015-67f700ecb66f" containerName="revoke-keys" Nov 04 01:23:54.578701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578664 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f139661f-18a5-432b-8b79-cfb1302f37d6" containerName="revoke-keys" Nov 04 01:23:54.578701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578669 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f139661f-18a5-432b-8b79-cfb1302f37d6" containerName="revoke-keys" Nov 04 01:23:54.578794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578705 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2f27d705-1a3c-44d3-99b2-126371818776" containerName="revoke-keys" Nov 04 01:23:54.578794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578711 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f27d705-1a3c-44d3-99b2-126371818776" containerName="revoke-keys" Nov 04 01:23:54.578856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578802 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe36d7d9-13ec-4ec7-b015-67f700ecb66f" containerName="revoke-keys" Nov 04 01:23:54.578856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578811 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f139661f-18a5-432b-8b79-cfb1302f37d6" containerName="revoke-keys" Nov 04 01:23:54.578856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.578823 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f27d705-1a3c-44d3-99b2-126371818776" containerName="revoke-keys" Nov 04 01:23:54.583037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.583015 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.592254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.591044 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:23:54.592254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.591168 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:23:54.592254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.591238 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:23:54.592254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.591263 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:23:54.592254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.592254 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g"] Nov 04 01:23:54.693724 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.693682 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.693724 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.693722 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.693959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.693936 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5gg\" (UniqueName: \"kubernetes.io/projected/9afc777e-5c73-4cd5-bc31-08ca02fc809a-kube-api-access-sp5gg\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.795754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.795716 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.795754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.795759 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.796005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.795976 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5gg\" (UniqueName: \"kubernetes.io/projected/9afc777e-5c73-4cd5-bc31-08ca02fc809a-kube-api-access-sp5gg\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.796527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.796506 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.798378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.798354 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.804496 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.804469 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5gg\" (UniqueName: \"kubernetes.io/projected/9afc777e-5c73-4cd5-bc31-08ca02fc809a-kube-api-access-sp5gg\") pod \"maas-api-revoke-sub-fa640c3a-w6q6g\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:54.908616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:54.908532 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:55.267564 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:23:55.267374 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afc777e_5c73_4cd5_bc31_08ca02fc809a.slice/crio-89ccdbeaf4e11ff6ba762d26a35b1485f132ce518c27197cbc943f49d6701c38 WatchSource:0}: Error finding container 89ccdbeaf4e11ff6ba762d26a35b1485f132ce518c27197cbc943f49d6701c38: Status 404 returned error can't find the container with id 89ccdbeaf4e11ff6ba762d26a35b1485f132ce518c27197cbc943f49d6701c38 Nov 04 01:23:55.269894 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:55.269743 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g"] Nov 04 01:23:55.339227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:55.339187 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" event={"ID":"9afc777e-5c73-4cd5-bc31-08ca02fc809a","Type":"ContainerStarted","Data":"89ccdbeaf4e11ff6ba762d26a35b1485f132ce518c27197cbc943f49d6701c38"} Nov 04 01:23:56.349564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:56.349528 5051 generic.go:356] "Generic (PLEG): container finished" podID="9afc777e-5c73-4cd5-bc31-08ca02fc809a" containerID="44f386153a7daed5c7d30f04d1b6d36ea2f3bfd125d104af8d877df3d020d5b8" exitCode=0 Nov 04 01:23:56.349994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:56.349613 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" event={"ID":"9afc777e-5c73-4cd5-bc31-08ca02fc809a","Type":"ContainerDied","Data":"44f386153a7daed5c7d30f04d1b6d36ea2f3bfd125d104af8d877df3d020d5b8"} Nov 04 01:23:57.404630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.404593 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt"] Nov 04 01:23:57.411001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.410951 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.422232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.419706 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt"] Nov 04 01:23:57.455673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.455640 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgrl\" (UniqueName: \"kubernetes.io/projected/b492feea-9e67-4e2a-aed9-8e9c3386e82a-kube-api-access-ghgrl\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.455854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.455784 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.455927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.455869 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.558372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.558334 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgrl\" (UniqueName: \"kubernetes.io/projected/b492feea-9e67-4e2a-aed9-8e9c3386e82a-kube-api-access-ghgrl\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.558571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.558440 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.558571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.558524 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.559433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.559407 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.562480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.562458 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.570633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.570610 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgrl\" (UniqueName: \"kubernetes.io/projected/b492feea-9e67-4e2a-aed9-8e9c3386e82a-kube-api-access-ghgrl\") pod \"maas-api-revoke-sub-19ee9690-tbbdt\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.599637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.599614 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:57.659864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.659827 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-xks-ca\") pod \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " Nov 04 01:23:57.660029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.659972 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afc777e-5c73-4cd5-bc31-08ca02fc809a-kube-api-access-sp5gg\" (UniqueName: \"kubernetes.io/projected/9afc777e-5c73-4cd5-bc31-08ca02fc809a-kube-api-access-sp5gg\") pod \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " Nov 04 01:23:57.660228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.660206 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-openshift-ca\") pod \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\" (UID: \"9afc777e-5c73-4cd5-bc31-08ca02fc809a\") " Nov 04 01:23:57.660754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.660705 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-openshift-ca" pod "9afc777e-5c73-4cd5-bc31-08ca02fc809a" (UID: "9afc777e-5c73-4cd5-bc31-08ca02fc809a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:23:57.660992 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.660968 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:57.662402 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.662373 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afc777e-5c73-4cd5-bc31-08ca02fc809a-kube-api-access-sp5gg" pod "9afc777e-5c73-4cd5-bc31-08ca02fc809a" (UID: "9afc777e-5c73-4cd5-bc31-08ca02fc809a"). InnerVolumeSpecName "kube-api-access-sp5gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:23:57.662663 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.662636 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-xks-ca" pod "9afc777e-5c73-4cd5-bc31-08ca02fc809a" (UID: "9afc777e-5c73-4cd5-bc31-08ca02fc809a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:23:57.737436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.737400 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:23:57.762820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.762786 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp5gg\" (UniqueName: \"kubernetes.io/projected/9afc777e-5c73-4cd5-bc31-08ca02fc809a-kube-api-access-sp5gg\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:57.762919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:57.762824 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9afc777e-5c73-4cd5-bc31-08ca02fc809a-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:23:58.037325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.037291 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts"] Nov 04 01:23:58.038016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.037999 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9afc777e-5c73-4cd5-bc31-08ca02fc809a" containerName="revoke-keys" Nov 04 01:23:58.038016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.038015 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afc777e-5c73-4cd5-bc31-08ca02fc809a" containerName="revoke-keys" Nov 04 01:23:58.038198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.038188 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afc777e-5c73-4cd5-bc31-08ca02fc809a" containerName="revoke-keys" Nov 04 01:23:58.042375 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.042354 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.096677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.096638 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts"] Nov 04 01:23:58.117561 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.117478 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.117785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.117650 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.117954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.117896 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bppjw\" (UniqueName: \"kubernetes.io/projected/9a7fe3ad-dbe8-42d9-bb41-722519b05069-kube-api-access-bppjw\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.134673 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:23:58.134644 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb492feea_9e67_4e2a_aed9_8e9c3386e82a.slice/crio-eafadf4b18dcaf0341c5e1ddf495d60dac2c491e0d472df212201b234c9854d1 WatchSource:0}: Error finding container eafadf4b18dcaf0341c5e1ddf495d60dac2c491e0d472df212201b234c9854d1: Status 404 returned error can't find the container with id eafadf4b18dcaf0341c5e1ddf495d60dac2c491e0d472df212201b234c9854d1 Nov 04 01:23:58.149541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.148750 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt"] Nov 04 01:23:58.219174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.219110 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.219297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.219265 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.219386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.219371 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bppjw\" (UniqueName: \"kubernetes.io/projected/9a7fe3ad-dbe8-42d9-bb41-722519b05069-kube-api-access-bppjw\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.219778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.219754 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.221703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.221683 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.240775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.240751 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bppjw\" (UniqueName: \"kubernetes.io/projected/9a7fe3ad-dbe8-42d9-bb41-722519b05069-kube-api-access-bppjw\") pod \"maas-api-revoke-sub-147c2eaf-rdtts\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.357745 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.357711 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:23:58.373047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.372977 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" event={"ID":"9afc777e-5c73-4cd5-bc31-08ca02fc809a","Type":"ContainerDied","Data":"89ccdbeaf4e11ff6ba762d26a35b1485f132ce518c27197cbc943f49d6701c38"} Nov 04 01:23:58.373047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.373002 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g" Nov 04 01:23:58.373047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.373042 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ccdbeaf4e11ff6ba762d26a35b1485f132ce518c27197cbc943f49d6701c38" Nov 04 01:23:58.374586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.374558 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" event={"ID":"b492feea-9e67-4e2a-aed9-8e9c3386e82a","Type":"ContainerStarted","Data":"eafadf4b18dcaf0341c5e1ddf495d60dac2c491e0d472df212201b234c9854d1"} Nov 04 01:23:58.673375 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:58.673345 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts"] Nov 04 01:23:59.387653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:59.387618 5051 generic.go:356] "Generic (PLEG): container finished" podID="b492feea-9e67-4e2a-aed9-8e9c3386e82a" containerID="221a742ac7d5b3cf0356ad06bf074207d9150031f2e349b0cce2e08120ccdff0" exitCode=0 Nov 04 01:23:59.387850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:59.387705 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" event={"ID":"b492feea-9e67-4e2a-aed9-8e9c3386e82a","Type":"ContainerDied","Data":"221a742ac7d5b3cf0356ad06bf074207d9150031f2e349b0cce2e08120ccdff0"} Nov 04 01:23:59.389683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:59.389650 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" event={"ID":"9a7fe3ad-dbe8-42d9-bb41-722519b05069","Type":"ContainerStarted","Data":"a48fee7b60e94ab75d08ef3539bbedccfe0f12ec0a896ed91a3a1767f89324c2"} Nov 04 01:23:59.389818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:59.389698 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" event={"ID":"9a7fe3ad-dbe8-42d9-bb41-722519b05069","Type":"ContainerStarted","Data":"4a234052f430858eb964d19a7f02a5eebbc1d22238b1f35cde801387db248922"} Nov 04 01:23:59.436573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:23:59.436518 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" podStartSLOduration=1.436505534 podStartE2EDuration="1.436505534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:23:59.431938384 +0000 UTC m=+1578.403687254" watchObservedRunningTime="2026-11-04 01:23:59.436505534 +0000 UTC m=+1578.408254404" Nov 04 01:24:00.406745 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.406709 5051 generic.go:356] "Generic (PLEG): container finished" podID="9a7fe3ad-dbe8-42d9-bb41-722519b05069" containerID="a48fee7b60e94ab75d08ef3539bbedccfe0f12ec0a896ed91a3a1767f89324c2" exitCode=0 Nov 04 01:24:00.407213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.406802 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" event={"ID":"9a7fe3ad-dbe8-42d9-bb41-722519b05069","Type":"ContainerDied","Data":"a48fee7b60e94ab75d08ef3539bbedccfe0f12ec0a896ed91a3a1767f89324c2"} Nov 04 01:24:00.685055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.684482 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:24:00.806777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.806735 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b492feea-9e67-4e2a-aed9-8e9c3386e82a-kube-api-access-ghgrl\" (UniqueName: \"kubernetes.io/projected/b492feea-9e67-4e2a-aed9-8e9c3386e82a-kube-api-access-ghgrl\") pod \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " Nov 04 01:24:00.841363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.839567 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-xks-ca\") pod \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " Nov 04 01:24:00.841363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.839814 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-openshift-ca\") pod \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\" (UID: \"b492feea-9e67-4e2a-aed9-8e9c3386e82a\") " Nov 04 01:24:00.913186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.911528 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-openshift-ca" pod "b492feea-9e67-4e2a-aed9-8e9c3386e82a" (UID: "b492feea-9e67-4e2a-aed9-8e9c3386e82a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:00.944182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.944079 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:00.968838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.963021 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-xks-ca" pod "b492feea-9e67-4e2a-aed9-8e9c3386e82a" (UID: "b492feea-9e67-4e2a-aed9-8e9c3386e82a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:00.968838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:00.963456 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b492feea-9e67-4e2a-aed9-8e9c3386e82a-kube-api-access-ghgrl" pod "b492feea-9e67-4e2a-aed9-8e9c3386e82a" (UID: "b492feea-9e67-4e2a-aed9-8e9c3386e82a"). InnerVolumeSpecName "kube-api-access-ghgrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:01.047145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.046732 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b492feea-9e67-4e2a-aed9-8e9c3386e82a-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:01.047145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.046776 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghgrl\" (UniqueName: \"kubernetes.io/projected/b492feea-9e67-4e2a-aed9-8e9c3386e82a-kube-api-access-ghgrl\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:01.442314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.437083 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" event={"ID":"b492feea-9e67-4e2a-aed9-8e9c3386e82a","Type":"ContainerDied","Data":"eafadf4b18dcaf0341c5e1ddf495d60dac2c491e0d472df212201b234c9854d1"} Nov 04 01:24:01.442314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.437200 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eafadf4b18dcaf0341c5e1ddf495d60dac2c491e0d472df212201b234c9854d1" Nov 04 01:24:01.442314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.437344 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt" Nov 04 01:24:01.830026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.829996 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:24:01.997323 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.997256 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-openshift-ca\") pod \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " Nov 04 01:24:01.997599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.997371 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-xks-ca\") pod \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " Nov 04 01:24:01.997599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.997534 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7fe3ad-dbe8-42d9-bb41-722519b05069-kube-api-access-bppjw\" (UniqueName: \"kubernetes.io/projected/9a7fe3ad-dbe8-42d9-bb41-722519b05069-kube-api-access-bppjw\") pod \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\" (UID: \"9a7fe3ad-dbe8-42d9-bb41-722519b05069\") " Nov 04 01:24:01.998211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.998177 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-openshift-ca" pod "9a7fe3ad-dbe8-42d9-bb41-722519b05069" (UID: "9a7fe3ad-dbe8-42d9-bb41-722519b05069"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:01.998871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.998844 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:01.999787 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:01.999758 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-xks-ca" pod "9a7fe3ad-dbe8-42d9-bb41-722519b05069" (UID: "9a7fe3ad-dbe8-42d9-bb41-722519b05069"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:02.000096 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.000071 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7fe3ad-dbe8-42d9-bb41-722519b05069-kube-api-access-bppjw" pod "9a7fe3ad-dbe8-42d9-bb41-722519b05069" (UID: "9a7fe3ad-dbe8-42d9-bb41-722519b05069"). InnerVolumeSpecName "kube-api-access-bppjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:02.100370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.100289 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bppjw\" (UniqueName: \"kubernetes.io/projected/9a7fe3ad-dbe8-42d9-bb41-722519b05069-kube-api-access-bppjw\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:02.100370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.100324 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7fe3ad-dbe8-42d9-bb41-722519b05069-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:02.283994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.283956 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d"] Nov 04 01:24:02.284723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.284704 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b492feea-9e67-4e2a-aed9-8e9c3386e82a" containerName="revoke-keys" Nov 04 01:24:02.284723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.284723 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b492feea-9e67-4e2a-aed9-8e9c3386e82a" containerName="revoke-keys" Nov 04 01:24:02.284853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.284740 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9a7fe3ad-dbe8-42d9-bb41-722519b05069" containerName="revoke-keys" Nov 04 01:24:02.284853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.284746 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7fe3ad-dbe8-42d9-bb41-722519b05069" containerName="revoke-keys" Nov 04 01:24:02.284970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.284883 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b492feea-9e67-4e2a-aed9-8e9c3386e82a" containerName="revoke-keys" Nov 04 01:24:02.284970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.284894 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7fe3ad-dbe8-42d9-bb41-722519b05069" containerName="revoke-keys" Nov 04 01:24:02.288993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.288971 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.298575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.298544 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d"] Nov 04 01:24:02.424280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.407945 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.424280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.408258 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4nsg\" (UniqueName: \"kubernetes.io/projected/2b994d45-a1e2-4058-a193-49b30619dfc3-kube-api-access-b4nsg\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.424280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.408386 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.468589 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.468551 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" event={"ID":"9a7fe3ad-dbe8-42d9-bb41-722519b05069","Type":"ContainerDied","Data":"4a234052f430858eb964d19a7f02a5eebbc1d22238b1f35cde801387db248922"} Nov 04 01:24:02.469197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.469103 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a234052f430858eb964d19a7f02a5eebbc1d22238b1f35cde801387db248922" Nov 04 01:24:02.469379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.469365 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts" Nov 04 01:24:02.510634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.510601 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4nsg\" (UniqueName: \"kubernetes.io/projected/2b994d45-a1e2-4058-a193-49b30619dfc3-kube-api-access-b4nsg\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.510766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.510733 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.510913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.510897 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.511853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.511822 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.513759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.513733 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.521484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.521434 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4nsg\" (UniqueName: \"kubernetes.io/projected/2b994d45-a1e2-4058-a193-49b30619dfc3-kube-api-access-b4nsg\") pod \"maas-api-revoke-sub-6a35b987-jmw8d\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.634338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.634295 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:02.932612 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:24:02.932584 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b994d45_a1e2_4058_a193_49b30619dfc3.slice/crio-2aa2d0ca850566ea1bb96bf3f4dc876cdf942b9ffc58e42c952a98a15909208c WatchSource:0}: Error finding container 2aa2d0ca850566ea1bb96bf3f4dc876cdf942b9ffc58e42c952a98a15909208c: Status 404 returned error can't find the container with id 2aa2d0ca850566ea1bb96bf3f4dc876cdf942b9ffc58e42c952a98a15909208c Nov 04 01:24:02.934769 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:02.934744 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d"] Nov 04 01:24:03.480550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:03.480513 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" event={"ID":"2b994d45-a1e2-4058-a193-49b30619dfc3","Type":"ContainerStarted","Data":"e20a2d5a67f23ed57885c77dac6f262bc02b52569d55e520269a7f85ab50cfec"} Nov 04 01:24:03.481011 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:03.480580 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" event={"ID":"2b994d45-a1e2-4058-a193-49b30619dfc3","Type":"ContainerStarted","Data":"2aa2d0ca850566ea1bb96bf3f4dc876cdf942b9ffc58e42c952a98a15909208c"} Nov 04 01:24:03.503220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:03.502801 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" podStartSLOduration=1.5027835920000001 podStartE2EDuration="1.502783592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:03.50164346 +0000 UTC m=+1582.473392331" watchObservedRunningTime="2026-11-04 01:24:03.502783592 +0000 UTC m=+1582.474532464" Nov 04 01:24:04.112462 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:24:04.112428 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:24:04.507167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:04.507105 5051 generic.go:356] "Generic (PLEG): container finished" podID="2b994d45-a1e2-4058-a193-49b30619dfc3" containerID="e20a2d5a67f23ed57885c77dac6f262bc02b52569d55e520269a7f85ab50cfec" exitCode=0 Nov 04 01:24:04.507628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:04.507190 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" event={"ID":"2b994d45-a1e2-4058-a193-49b30619dfc3","Type":"ContainerDied","Data":"e20a2d5a67f23ed57885c77dac6f262bc02b52569d55e520269a7f85ab50cfec"} Nov 04 01:24:05.841577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.841514 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:05.889382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.889346 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-xks-ca\") pod \"2b994d45-a1e2-4058-a193-49b30619dfc3\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " Nov 04 01:24:05.889559 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.889474 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-openshift-ca\") pod \"2b994d45-a1e2-4058-a193-49b30619dfc3\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " Nov 04 01:24:05.889728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.889705 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b994d45-a1e2-4058-a193-49b30619dfc3-kube-api-access-b4nsg\" (UniqueName: \"kubernetes.io/projected/2b994d45-a1e2-4058-a193-49b30619dfc3-kube-api-access-b4nsg\") pod \"2b994d45-a1e2-4058-a193-49b30619dfc3\" (UID: \"2b994d45-a1e2-4058-a193-49b30619dfc3\") " Nov 04 01:24:05.890482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.890447 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-openshift-ca" pod "2b994d45-a1e2-4058-a193-49b30619dfc3" (UID: "2b994d45-a1e2-4058-a193-49b30619dfc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:05.890594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.890569 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:05.892593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.892563 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-xks-ca" pod "2b994d45-a1e2-4058-a193-49b30619dfc3" (UID: "2b994d45-a1e2-4058-a193-49b30619dfc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:05.895764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.895726 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b994d45-a1e2-4058-a193-49b30619dfc3-kube-api-access-b4nsg" pod "2b994d45-a1e2-4058-a193-49b30619dfc3" (UID: "2b994d45-a1e2-4058-a193-49b30619dfc3"). InnerVolumeSpecName "kube-api-access-b4nsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:05.992278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.992240 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b994d45-a1e2-4058-a193-49b30619dfc3-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:05.992278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:05.992279 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4nsg\" (UniqueName: \"kubernetes.io/projected/2b994d45-a1e2-4058-a193-49b30619dfc3-kube-api-access-b4nsg\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:06.590363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.590329 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" Nov 04 01:24:06.590363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.590346 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d" event={"ID":"2b994d45-a1e2-4058-a193-49b30619dfc3","Type":"ContainerDied","Data":"2aa2d0ca850566ea1bb96bf3f4dc876cdf942b9ffc58e42c952a98a15909208c"} Nov 04 01:24:06.590575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.590393 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aa2d0ca850566ea1bb96bf3f4dc876cdf942b9ffc58e42c952a98a15909208c" Nov 04 01:24:06.913499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.913424 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr"] Nov 04 01:24:06.914238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.914217 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2b994d45-a1e2-4058-a193-49b30619dfc3" containerName="revoke-keys" Nov 04 01:24:06.914306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.914241 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b994d45-a1e2-4058-a193-49b30619dfc3" containerName="revoke-keys" Nov 04 01:24:06.914411 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.914401 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b994d45-a1e2-4058-a193-49b30619dfc3" containerName="revoke-keys" Nov 04 01:24:06.918482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.918463 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:06.924982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.924955 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:24:06.924982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.924967 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:24:06.925150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.925019 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:24:06.925150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.925040 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:24:06.927563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:06.927540 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr"] Nov 04 01:24:07.012279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.012237 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.012461 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.012379 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.012867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.012673 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7msg\" (UniqueName: \"kubernetes.io/projected/bc53491e-7a47-49b1-86b3-d47440df1135-kube-api-access-z7msg\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.114369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.114335 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7msg\" (UniqueName: \"kubernetes.io/projected/bc53491e-7a47-49b1-86b3-d47440df1135-kube-api-access-z7msg\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.114520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.114494 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.114570 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.114560 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.115183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.115159 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.117134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.117096 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.128839 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.128810 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7msg\" (UniqueName: \"kubernetes.io/projected/bc53491e-7a47-49b1-86b3-d47440df1135-kube-api-access-z7msg\") pod \"maas-api-revoke-sub-18e42b7f-nhjdr\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.238438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.238411 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:07.601177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.601054 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr"] Nov 04 01:24:07.607677 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:24:07.607643 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc53491e_7a47_49b1_86b3_d47440df1135.slice/crio-5a3004a51779ba16440ca032efe139573ecb438f2e1de75c77932242218c5b1b WatchSource:0}: Error finding container 5a3004a51779ba16440ca032efe139573ecb438f2e1de75c77932242218c5b1b: Status 404 returned error can't find the container with id 5a3004a51779ba16440ca032efe139573ecb438f2e1de75c77932242218c5b1b Nov 04 01:24:07.894193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.880170 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j"] Nov 04 01:24:07.900164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.897227 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:07.949276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.948743 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:07.949276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.948918 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5gx4\" (UniqueName: \"kubernetes.io/projected/36359561-fa5d-4c84-8e13-bbd1ab48ab40-kube-api-access-b5gx4\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:07.949276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.949042 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:07.965905 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:07.959968 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j"] Nov 04 01:24:08.056107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:08.056055 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5gx4\" (UniqueName: \"kubernetes.io/projected/36359561-fa5d-4c84-8e13-bbd1ab48ab40-kube-api-access-b5gx4\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:08.056319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:08.056192 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:08.060148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:08.056470 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:08.060148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:08.057350 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:08.143599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:08.135667 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5gx4\" (UniqueName: \"kubernetes.io/projected/36359561-fa5d-4c84-8e13-bbd1ab48ab40-kube-api-access-b5gx4\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:08.189217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:08.179648 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d8aa4aa-wh85j\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:08.313143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:08.302899 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:08.858144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:08.824320 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:09.054183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.027776 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:09.054183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.031242 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.106555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.102009 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" event={"ID":"bc53491e-7a47-49b1-86b3-d47440df1135","Type":"ContainerStarted","Data":"5a3004a51779ba16440ca032efe139573ecb438f2e1de75c77932242218c5b1b"} Nov 04 01:24:09.317875 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.317217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.317875 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.317524 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:24:09.376809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.375275 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.527146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.527067 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.593268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.593219 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:09.828389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.828343 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:09.828851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:09.828822 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:24:10.215252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.215197 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:10.220987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.220949 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:24:10.527774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.239656 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:10.576146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.267562 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.576585 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.321114 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.611279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.587569 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:24:10.611279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.587637 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:24:10.611279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.591523 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:10.611279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.594400 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.611279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.598826 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:24:10.637150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.633010 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.778633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.579018 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:10.779588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.779562 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:24:10.822158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.817810 5051 generic.go:356] "Generic (PLEG): container finished" podID="bc53491e-7a47-49b1-86b3-d47440df1135" containerID="0aff91e2e5ab7a21544f585205918ee519dbe7dc4f61879e33a9cc416f7a480a" exitCode=0 Nov 04 01:24:10.822158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.817884 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" event={"ID":"bc53491e-7a47-49b1-86b3-d47440df1135","Type":"ContainerDied","Data":"0aff91e2e5ab7a21544f585205918ee519dbe7dc4f61879e33a9cc416f7a480a"} Nov 04 01:24:10.886244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:10.886195 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.245144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:11.241291 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.603207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:11.562438 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.628147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:11.608980 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:24:11.667752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:11.667696 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:11.697315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:11.667993 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.697695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:11.672272 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:11.836794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:11.836689 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:11.929190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:11.919616 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:12.689167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:12.688642 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:12.865029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:12.837587 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:13.029269 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.029223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.104729 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:24:13.067013 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.767s" Nov 04 01:24:13.104729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.099568 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:24:13.104729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.099626 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:24:13.104729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.099680 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:13.200146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.136249 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.200146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.156994 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.245152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.233271 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:13.245152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.233340 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.697645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.697599 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:13.705522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.698537 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:13.706099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.706067 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:13.957144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.712190 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:13.963173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.771805 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:13.963653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.903305 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:13.996146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:13.985564 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" event={"ID":"36359561-fa5d-4c84-8e13-bbd1ab48ab40","Type":"ContainerStarted","Data":"43a11bfa2a8c8c555c9384f22a571b7338b1b82c97463af4381dd60aad82a8c1"} Nov 04 01:24:14.021462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.021359 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:14.021462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.021443 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.032312 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.032274 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.102214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.102168 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.278154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.271230 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:14.278154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.271297 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.699402 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.699283 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.708245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.708202 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:14.725969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.725933 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" event={"ID":"36359561-fa5d-4c84-8e13-bbd1ab48ab40","Type":"ContainerStarted","Data":"bcc72b4f33e598e90a04b40058f0c9e088f9ccd647ff60dfd2f78398bb9ae43f"} Nov 04 01:24:14.824847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.823679 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j"] Nov 04 01:24:14.977203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.960099 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.006774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.961296 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:15.030146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.979264 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.030146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:14.979300 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:15.046178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.042740 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.132334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.113227 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.147147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.145874 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" podStartSLOduration=8.145851313 podStartE2EDuration="8.145851313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:14.823649663 +0000 UTC m=+1593.795398534" watchObservedRunningTime="2026-11-04 01:24:15.145851313 +0000 UTC m=+1594.117600184" Nov 04 01:24:15.332142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.310409 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:15.332142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.310673 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.715197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.708581 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.715197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.711913 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.855145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.851807 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:15.885482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.885424 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:15.885783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:15.885749 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.031208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.009181 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:16.031208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.011007 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:16.031208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.011404 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:16.032872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.032825 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.051636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.047663 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:16.105803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.105750 5051 generic.go:356] "Generic (PLEG): container finished" podID="36359561-fa5d-4c84-8e13-bbd1ab48ab40" containerID="bcc72b4f33e598e90a04b40058f0c9e088f9ccd647ff60dfd2f78398bb9ae43f" exitCode=0 Nov 04 01:24:16.180153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.106150 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" event={"ID":"36359561-fa5d-4c84-8e13-bbd1ab48ab40","Type":"ContainerDied","Data":"bcc72b4f33e598e90a04b40058f0c9e088f9ccd647ff60dfd2f78398bb9ae43f"} Nov 04 01:24:16.181136 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.120483 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:16.181556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.150560 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:16.181719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.181692 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.720290 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.720241 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:16.796346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.781277 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:16.811397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.781959 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:16.811397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:16.807672 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:17.106145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.099246 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:17.106145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.099635 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:17.106145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.099931 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:17.106145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.100599 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:17.106145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.100636 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:17.240145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.231509 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:17.334669 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.307102 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Nov 04 01:24:17.391258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.372405 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:17.412534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.412497 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:17.450142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.447264 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:17.524149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.371223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:17.845283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.845233 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:17.896166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.862609 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:17.934161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:17.933439 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr"] Nov 04 01:24:18.016756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.016724 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:18.016951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.016924 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.080629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.080597 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr"] Nov 04 01:24:18.080998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.080981 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.099610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.099514 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.115447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.103911 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.142295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.103939 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:18.190960 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.164159 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8sd\" (UniqueName: \"kubernetes.io/projected/557acf14-07aa-4b15-9ce8-6e48c960755e-kube-api-access-nn8sd\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.191305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.191272 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.217148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.191672 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.280272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.280222 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.337145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.320939 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8sd\" (UniqueName: \"kubernetes.io/projected/557acf14-07aa-4b15-9ce8-6e48c960755e-kube-api-access-nn8sd\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.337519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.337482 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.337898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.337873 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.353213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.347370 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.401212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.353469 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:18.401511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.401472 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.416146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.405825 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.462147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.436381 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:18.491147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.468723 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.491147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.470369 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8sd\" (UniqueName: \"kubernetes.io/projected/557acf14-07aa-4b15-9ce8-6e48c960755e-kube-api-access-nn8sd\") pod \"maas-api-revoke-sub-91f6ff71-z4nqr\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.508147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.496722 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.508474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.508444 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:18.508636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.508612 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.552146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.535218 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:18.615197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.612927 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc53491e-7a47-49b1-86b3-d47440df1135-kube-api-access-z7msg\" (UniqueName: \"kubernetes.io/projected/bc53491e-7a47-49b1-86b3-d47440df1135-kube-api-access-z7msg\") pod \"bc53491e-7a47-49b1-86b3-d47440df1135\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " Nov 04 01:24:18.615197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.613103 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-xks-ca\") pod \"bc53491e-7a47-49b1-86b3-d47440df1135\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " Nov 04 01:24:18.615197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.613406 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-openshift-ca\") pod \"bc53491e-7a47-49b1-86b3-d47440df1135\" (UID: \"bc53491e-7a47-49b1-86b3-d47440df1135\") " Nov 04 01:24:18.700146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.672409 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-openshift-ca" pod "bc53491e-7a47-49b1-86b3-d47440df1135" (UID: "bc53491e-7a47-49b1-86b3-d47440df1135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:18.803597 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.802355 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:18.812189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.807902 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:18.866414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.866212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:18.909862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.909812 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:18.962842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.959086 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-xks-ca" pod "bc53491e-7a47-49b1-86b3-d47440df1135" (UID: "bc53491e-7a47-49b1-86b3-d47440df1135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:18.964236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.964180 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" event={"ID":"bc53491e-7a47-49b1-86b3-d47440df1135","Type":"ContainerDied","Data":"5a3004a51779ba16440ca032efe139573ecb438f2e1de75c77932242218c5b1b"} Nov 04 01:24:18.964676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.964652 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a3004a51779ba16440ca032efe139573ecb438f2e1de75c77932242218c5b1b" Nov 04 01:24:18.964818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.964491 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr" Nov 04 01:24:18.991585 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.991526 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc53491e-7a47-49b1-86b3-d47440df1135-kube-api-access-z7msg" pod "bc53491e-7a47-49b1-86b3-d47440df1135" (UID: "bc53491e-7a47-49b1-86b3-d47440df1135"). InnerVolumeSpecName "kube-api-access-z7msg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:18.992832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:18.992804 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7msg\" (UniqueName: \"kubernetes.io/projected/bc53491e-7a47-49b1-86b3-d47440df1135-kube-api-access-z7msg\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:19.009921 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.009892 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc53491e-7a47-49b1-86b3-d47440df1135-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:19.138513 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.117886 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:19.241147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.212303 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:19.241147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.236833 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:19.252147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.241497 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:19.252147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.241559 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:19.283150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.279674 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:19.308440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.307275 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:19.358423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.358238 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:19.358423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.358303 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:19.417436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.416243 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:19.417436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.416345 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:19.557224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.556215 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:19.924012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.896515 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:19.924012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:19.923478 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:20.221284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.221181 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:20.340776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.310400 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:20.340776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.318650 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:20.340776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.323516 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:20.454157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.432701 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:20.601206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.587191 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:20.765166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.757704 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Nov 04 01:24:20.966416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.966297 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:21.115096 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:20.993787 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:21.296216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:21.225430 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:21.328162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:21.328097 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:21.329350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:21.329317 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:21.361219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:21.361167 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:21.678523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:21.678477 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Nov 04 01:24:21.678793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:21.611296 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:24:21.678893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:21.628664 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:21.698800 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:21.611226 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:22.208495 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:24:22.208465 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.023s" Nov 04 01:24:22.208983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.208964 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr"] Nov 04 01:24:22.209239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.209224 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:24:22.209656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.209640 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:22.209813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.209703 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" event={"ID":"36359561-fa5d-4c84-8e13-bbd1ab48ab40","Type":"ContainerDied","Data":"43a11bfa2a8c8c555c9384f22a571b7338b1b82c97463af4381dd60aad82a8c1"} Nov 04 01:24:22.209946 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.209934 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43a11bfa2a8c8c555c9384f22a571b7338b1b82c97463af4381dd60aad82a8c1" Nov 04 01:24:22.217164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.216887 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:24:22.218541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.218515 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:24:22.222919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.222892 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:24:22.223558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.223536 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:24:22.225036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.225012 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" event={"ID":"557acf14-07aa-4b15-9ce8-6e48c960755e","Type":"ContainerStarted","Data":"bb4ab7c6709cb2da3ac4740f26f7bab86299b01f0e05716f7b7e7d76823c6dd7"} Nov 04 01:24:22.237948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.236766 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:24:22.237948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.237490 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:24:22.244009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.243072 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:24:22.322325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.322297 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-xks-ca\") pod \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " Nov 04 01:24:22.322718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.322701 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-openshift-ca\") pod \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " Nov 04 01:24:22.322871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.322860 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36359561-fa5d-4c84-8e13-bbd1ab48ab40-kube-api-access-b5gx4\" (UniqueName: \"kubernetes.io/projected/36359561-fa5d-4c84-8e13-bbd1ab48ab40-kube-api-access-b5gx4\") pod \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\" (UID: \"36359561-fa5d-4c84-8e13-bbd1ab48ab40\") " Nov 04 01:24:22.326194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.326154 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-openshift-ca" pod "36359561-fa5d-4c84-8e13-bbd1ab48ab40" (UID: "36359561-fa5d-4c84-8e13-bbd1ab48ab40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:22.327551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.327509 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36359561-fa5d-4c84-8e13-bbd1ab48ab40-kube-api-access-b5gx4" pod "36359561-fa5d-4c84-8e13-bbd1ab48ab40" (UID: "36359561-fa5d-4c84-8e13-bbd1ab48ab40"). InnerVolumeSpecName "kube-api-access-b5gx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:22.330051 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.330026 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-xks-ca" pod "36359561-fa5d-4c84-8e13-bbd1ab48ab40" (UID: "36359561-fa5d-4c84-8e13-bbd1ab48ab40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:22.432005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.427758 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:22.432005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.427792 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5gx4\" (UniqueName: \"kubernetes.io/projected/36359561-fa5d-4c84-8e13-bbd1ab48ab40-kube-api-access-b5gx4\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:22.432005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:22.427808 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36359561-fa5d-4c84-8e13-bbd1ab48ab40-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:23.006801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.006767 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm"] Nov 04 01:24:23.007722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.007702 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bc53491e-7a47-49b1-86b3-d47440df1135" containerName="revoke-keys" Nov 04 01:24:23.007722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.007722 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc53491e-7a47-49b1-86b3-d47440df1135" containerName="revoke-keys" Nov 04 01:24:23.007866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.007745 5051 cpu_manager.go:414] "containerMap: removing container" podUID="36359561-fa5d-4c84-8e13-bbd1ab48ab40" containerName="revoke-keys" Nov 04 01:24:23.007866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.007755 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36359561-fa5d-4c84-8e13-bbd1ab48ab40" containerName="revoke-keys" Nov 04 01:24:23.007951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.007896 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc53491e-7a47-49b1-86b3-d47440df1135" containerName="revoke-keys" Nov 04 01:24:23.007951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.007910 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="36359561-fa5d-4c84-8e13-bbd1ab48ab40" containerName="revoke-keys" Nov 04 01:24:23.012195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.012174 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.027953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.027927 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm"] Nov 04 01:24:23.145381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.145347 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.145555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.145501 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.145752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.145732 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgw2h\" (UniqueName: \"kubernetes.io/projected/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-kube-api-access-hgw2h\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.276195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.267550 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgw2h\" (UniqueName: \"kubernetes.io/projected/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-kube-api-access-hgw2h\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.276195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.267711 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.276195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.267853 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.296265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.294208 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.313143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.308594 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.327940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.327895 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgw2h\" (UniqueName: \"kubernetes.io/projected/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-kube-api-access-hgw2h\") pod \"maas-api-revoke-sub-65d4ea46-4fvpm\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.341672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.340461 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:23.349737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.349704 5051 generic.go:356] "Generic (PLEG): container finished" podID="557acf14-07aa-4b15-9ce8-6e48c960755e" containerID="6d2a9f5c61bf1c17243ee62457b3d9595b17f6a38b81e99e65756ddf0871f3c6" exitCode=0 Nov 04 01:24:23.350023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.349881 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j" Nov 04 01:24:23.351080 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.351041 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" event={"ID":"557acf14-07aa-4b15-9ce8-6e48c960755e","Type":"ContainerDied","Data":"6d2a9f5c61bf1c17243ee62457b3d9595b17f6a38b81e99e65756ddf0871f3c6"} Nov 04 01:24:23.655282 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:24:23.655250 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bf7c0d_6fcd_46a0_80f4_cd39c247e483.slice/crio-b1e28238738ac8948a6675092709599d84c71ea77c53a0e01df0a46e0bbef545 WatchSource:0}: Error finding container b1e28238738ac8948a6675092709599d84c71ea77c53a0e01df0a46e0bbef545: Status 404 returned error can't find the container with id b1e28238738ac8948a6675092709599d84c71ea77c53a0e01df0a46e0bbef545 Nov 04 01:24:23.658401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.658378 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm"] Nov 04 01:24:23.718299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.718266 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd"] Nov 04 01:24:23.726027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.726003 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:23.745992 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.745960 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd"] Nov 04 01:24:23.890650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.890275 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:23.890650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.890639 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8zk\" (UniqueName: \"kubernetes.io/projected/02116ac0-a24a-4c8f-a87c-e206df67e1ee-kube-api-access-kd8zk\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:23.890872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.890724 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:23.992596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.992560 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8zk\" (UniqueName: \"kubernetes.io/projected/02116ac0-a24a-4c8f-a87c-e206df67e1ee-kube-api-access-kd8zk\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:23.992779 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.992666 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:23.992779 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.992729 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:23.993506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.993481 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:23.995358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:23.995332 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:24.004297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.004272 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8zk\" (UniqueName: \"kubernetes.io/projected/02116ac0-a24a-4c8f-a87c-e206df67e1ee-kube-api-access-kd8zk\") pod \"maas-api-revoke-sub-40e43d6e-452vd\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:24.050983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.050954 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:24.351347 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:24:24.351314 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02116ac0_a24a_4c8f_a87c_e206df67e1ee.slice/crio-52ba768631733ae7179db2f7bd2a2d9d39f8a833415aa737dce22f6af09e28e0 WatchSource:0}: Error finding container 52ba768631733ae7179db2f7bd2a2d9d39f8a833415aa737dce22f6af09e28e0: Status 404 returned error can't find the container with id 52ba768631733ae7179db2f7bd2a2d9d39f8a833415aa737dce22f6af09e28e0 Nov 04 01:24:24.354997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.354869 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd"] Nov 04 01:24:24.373694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.373658 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" event={"ID":"88bf7c0d-6fcd-46a0-80f4-cd39c247e483","Type":"ContainerStarted","Data":"c9c6fb874d1b0f3abbc14883ff7da274610d5544379e252ed8e192f6eb3747dd"} Nov 04 01:24:24.374300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.374269 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" event={"ID":"88bf7c0d-6fcd-46a0-80f4-cd39c247e483","Type":"ContainerStarted","Data":"b1e28238738ac8948a6675092709599d84c71ea77c53a0e01df0a46e0bbef545"} Nov 04 01:24:24.420438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.420325 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" podStartSLOduration=2.42030784 podStartE2EDuration="2.42030784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:24.417618095 +0000 UTC m=+1603.389366970" watchObservedRunningTime="2026-11-04 01:24:24.42030784 +0000 UTC m=+1603.392056714" Nov 04 01:24:24.733720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.733696 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:24.787886 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.787841 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-xks-ca\") pod \"557acf14-07aa-4b15-9ce8-6e48c960755e\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " Nov 04 01:24:24.788099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.788079 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-openshift-ca\") pod \"557acf14-07aa-4b15-9ce8-6e48c960755e\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " Nov 04 01:24:24.788234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.788214 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557acf14-07aa-4b15-9ce8-6e48c960755e-kube-api-access-nn8sd\" (UniqueName: \"kubernetes.io/projected/557acf14-07aa-4b15-9ce8-6e48c960755e-kube-api-access-nn8sd\") pod \"557acf14-07aa-4b15-9ce8-6e48c960755e\" (UID: \"557acf14-07aa-4b15-9ce8-6e48c960755e\") " Nov 04 01:24:24.788763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.788729 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-openshift-ca" pod "557acf14-07aa-4b15-9ce8-6e48c960755e" (UID: "557acf14-07aa-4b15-9ce8-6e48c960755e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:24.789444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.789424 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:24.850018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.849981 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-xks-ca" pod "557acf14-07aa-4b15-9ce8-6e48c960755e" (UID: "557acf14-07aa-4b15-9ce8-6e48c960755e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:24.850319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.850295 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557acf14-07aa-4b15-9ce8-6e48c960755e-kube-api-access-nn8sd" pod "557acf14-07aa-4b15-9ce8-6e48c960755e" (UID: "557acf14-07aa-4b15-9ce8-6e48c960755e"). InnerVolumeSpecName "kube-api-access-nn8sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:24.891805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.891753 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/557acf14-07aa-4b15-9ce8-6e48c960755e-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:24.891805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:24.891805 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn8sd\" (UniqueName: \"kubernetes.io/projected/557acf14-07aa-4b15-9ce8-6e48c960755e-kube-api-access-nn8sd\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:25.436541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:25.436214 5051 generic.go:356] "Generic (PLEG): container finished" podID="88bf7c0d-6fcd-46a0-80f4-cd39c247e483" containerID="c9c6fb874d1b0f3abbc14883ff7da274610d5544379e252ed8e192f6eb3747dd" exitCode=0 Nov 04 01:24:25.436541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:25.436378 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" event={"ID":"88bf7c0d-6fcd-46a0-80f4-cd39c247e483","Type":"ContainerDied","Data":"c9c6fb874d1b0f3abbc14883ff7da274610d5544379e252ed8e192f6eb3747dd"} Nov 04 01:24:25.439305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:25.439251 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" Nov 04 01:24:25.440514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:25.440487 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr" event={"ID":"557acf14-07aa-4b15-9ce8-6e48c960755e","Type":"ContainerDied","Data":"bb4ab7c6709cb2da3ac4740f26f7bab86299b01f0e05716f7b7e7d76823c6dd7"} Nov 04 01:24:25.440688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:25.440670 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb4ab7c6709cb2da3ac4740f26f7bab86299b01f0e05716f7b7e7d76823c6dd7" Nov 04 01:24:25.445018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:25.444982 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" event={"ID":"02116ac0-a24a-4c8f-a87c-e206df67e1ee","Type":"ContainerStarted","Data":"d0707e1e40bbf5f177ed9782d938ab50e9ff2ef55f010a9b5e3c4e8f7194ed01"} Nov 04 01:24:25.445204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:25.445043 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" event={"ID":"02116ac0-a24a-4c8f-a87c-e206df67e1ee","Type":"ContainerStarted","Data":"52ba768631733ae7179db2f7bd2a2d9d39f8a833415aa737dce22f6af09e28e0"} Nov 04 01:24:25.497025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:25.496973 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" podStartSLOduration=2.496957673 podStartE2EDuration="2.496957673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:25.493383177 +0000 UTC m=+1604.465132048" watchObservedRunningTime="2026-11-04 01:24:25.496957673 +0000 UTC m=+1604.468706534" Nov 04 01:24:26.456081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.456038 5051 generic.go:356] "Generic (PLEG): container finished" podID="02116ac0-a24a-4c8f-a87c-e206df67e1ee" containerID="d0707e1e40bbf5f177ed9782d938ab50e9ff2ef55f010a9b5e3c4e8f7194ed01" exitCode=0 Nov 04 01:24:26.456081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.456062 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" event={"ID":"02116ac0-a24a-4c8f-a87c-e206df67e1ee","Type":"ContainerDied","Data":"d0707e1e40bbf5f177ed9782d938ab50e9ff2ef55f010a9b5e3c4e8f7194ed01"} Nov 04 01:24:26.669916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.669889 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:26.735455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.735346 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-openshift-ca\") pod \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " Nov 04 01:24:26.735651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.735519 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-kube-api-access-hgw2h\" (UniqueName: \"kubernetes.io/projected/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-kube-api-access-hgw2h\") pod \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " Nov 04 01:24:26.735651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.735620 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-xks-ca\") pod \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\" (UID: \"88bf7c0d-6fcd-46a0-80f4-cd39c247e483\") " Nov 04 01:24:26.735783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.735750 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-openshift-ca" pod "88bf7c0d-6fcd-46a0-80f4-cd39c247e483" (UID: "88bf7c0d-6fcd-46a0-80f4-cd39c247e483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:26.736530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.736462 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:26.738183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.738156 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-kube-api-access-hgw2h" pod "88bf7c0d-6fcd-46a0-80f4-cd39c247e483" (UID: "88bf7c0d-6fcd-46a0-80f4-cd39c247e483"). InnerVolumeSpecName "kube-api-access-hgw2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:26.738441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.738419 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-xks-ca" pod "88bf7c0d-6fcd-46a0-80f4-cd39c247e483" (UID: "88bf7c0d-6fcd-46a0-80f4-cd39c247e483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:26.806053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.806017 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd"] Nov 04 01:24:26.806821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.806799 5051 cpu_manager.go:414] "containerMap: removing container" podUID="88bf7c0d-6fcd-46a0-80f4-cd39c247e483" containerName="revoke-keys" Nov 04 01:24:26.806924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.806824 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88bf7c0d-6fcd-46a0-80f4-cd39c247e483" containerName="revoke-keys" Nov 04 01:24:26.806924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.806846 5051 cpu_manager.go:414] "containerMap: removing container" podUID="557acf14-07aa-4b15-9ce8-6e48c960755e" containerName="revoke-keys" Nov 04 01:24:26.806924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.806853 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557acf14-07aa-4b15-9ce8-6e48c960755e" containerName="revoke-keys" Nov 04 01:24:26.807077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.807043 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="88bf7c0d-6fcd-46a0-80f4-cd39c247e483" containerName="revoke-keys" Nov 04 01:24:26.807077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.807057 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="557acf14-07aa-4b15-9ce8-6e48c960755e" containerName="revoke-keys" Nov 04 01:24:26.811264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.811238 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:26.831344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.831307 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd"] Nov 04 01:24:26.849016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.840353 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgw2h\" (UniqueName: \"kubernetes.io/projected/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-kube-api-access-hgw2h\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:26.849016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.840392 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88bf7c0d-6fcd-46a0-80f4-cd39c247e483-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:26.941475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.941435 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:26.941681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.941493 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:26.941797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:26.941777 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4bc\" (UniqueName: \"kubernetes.io/projected/1c0a58ec-6d24-4037-973b-4450e76ce371-kube-api-access-sg4bc\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:27.043344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.043252 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4bc\" (UniqueName: \"kubernetes.io/projected/1c0a58ec-6d24-4037-973b-4450e76ce371-kube-api-access-sg4bc\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:27.043529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.043511 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:27.043586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.043572 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:27.044212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.044185 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:27.046058 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.046037 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:27.051691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.051668 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4bc\" (UniqueName: \"kubernetes.io/projected/1c0a58ec-6d24-4037-973b-4450e76ce371-kube-api-access-sg4bc\") pod \"maas-api-revoke-sub-c5c4e262-rjdsd\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:27.132704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.132664 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:27.460529 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:24:27.460305 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0a58ec_6d24_4037_973b_4450e76ce371.slice/crio-dc15b7ae5fbe7f6d72ba93d8221e2870357e2cf7c070fda16f5592c04a1c711e WatchSource:0}: Error finding container dc15b7ae5fbe7f6d72ba93d8221e2870357e2cf7c070fda16f5592c04a1c711e: Status 404 returned error can't find the container with id dc15b7ae5fbe7f6d72ba93d8221e2870357e2cf7c070fda16f5592c04a1c711e Nov 04 01:24:27.475946 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.475542 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd"] Nov 04 01:24:27.481788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.481763 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" Nov 04 01:24:27.481914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.481790 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm" event={"ID":"88bf7c0d-6fcd-46a0-80f4-cd39c247e483","Type":"ContainerDied","Data":"b1e28238738ac8948a6675092709599d84c71ea77c53a0e01df0a46e0bbef545"} Nov 04 01:24:27.481914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.481844 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e28238738ac8948a6675092709599d84c71ea77c53a0e01df0a46e0bbef545" Nov 04 01:24:27.483629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.483603 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" event={"ID":"1c0a58ec-6d24-4037-973b-4450e76ce371","Type":"ContainerStarted","Data":"dc15b7ae5fbe7f6d72ba93d8221e2870357e2cf7c070fda16f5592c04a1c711e"} Nov 04 01:24:27.676526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.676496 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:27.783972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.783924 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-openshift-ca\") pod \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " Nov 04 01:24:27.784199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.784178 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-xks-ca\") pod \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " Nov 04 01:24:27.784342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.784314 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02116ac0-a24a-4c8f-a87c-e206df67e1ee-kube-api-access-kd8zk\" (UniqueName: \"kubernetes.io/projected/02116ac0-a24a-4c8f-a87c-e206df67e1ee-kube-api-access-kd8zk\") pod \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\" (UID: \"02116ac0-a24a-4c8f-a87c-e206df67e1ee\") " Nov 04 01:24:27.784607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.784578 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-openshift-ca" pod "02116ac0-a24a-4c8f-a87c-e206df67e1ee" (UID: "02116ac0-a24a-4c8f-a87c-e206df67e1ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:27.784995 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.784972 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:27.786591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.786561 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-xks-ca" pod "02116ac0-a24a-4c8f-a87c-e206df67e1ee" (UID: "02116ac0-a24a-4c8f-a87c-e206df67e1ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:27.786985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.786963 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02116ac0-a24a-4c8f-a87c-e206df67e1ee-kube-api-access-kd8zk" pod "02116ac0-a24a-4c8f-a87c-e206df67e1ee" (UID: "02116ac0-a24a-4c8f-a87c-e206df67e1ee"). InnerVolumeSpecName "kube-api-access-kd8zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:27.887017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.886976 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd8zk\" (UniqueName: \"kubernetes.io/projected/02116ac0-a24a-4c8f-a87c-e206df67e1ee-kube-api-access-kd8zk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:27.887017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:27.887015 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02116ac0-a24a-4c8f-a87c-e206df67e1ee-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:28.494752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:28.494720 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:24:28.495249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:28.494721 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" event={"ID":"02116ac0-a24a-4c8f-a87c-e206df67e1ee","Type":"ContainerDied","Data":"52ba768631733ae7179db2f7bd2a2d9d39f8a833415aa737dce22f6af09e28e0"} Nov 04 01:24:28.495249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:28.494863 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ba768631733ae7179db2f7bd2a2d9d39f8a833415aa737dce22f6af09e28e0" Nov 04 01:24:28.496856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:28.496830 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" event={"ID":"1c0a58ec-6d24-4037-973b-4450e76ce371","Type":"ContainerStarted","Data":"b1d51d57af98fe7f586a0076f7a9c320074c38b8673bdd249e5c9c2cbf0d2197"} Nov 04 01:24:28.526466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:28.526415 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" podStartSLOduration=2.526400499 podStartE2EDuration="2.526400499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:28.519448714 +0000 UTC m=+1607.491197599" watchObservedRunningTime="2026-11-04 01:24:28.526400499 +0000 UTC m=+1607.498149390" Nov 04 01:24:29.520435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.520401 5051 generic.go:356] "Generic (PLEG): container finished" podID="1c0a58ec-6d24-4037-973b-4450e76ce371" containerID="b1d51d57af98fe7f586a0076f7a9c320074c38b8673bdd249e5c9c2cbf0d2197" exitCode=0 Nov 04 01:24:29.520924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.520486 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" event={"ID":"1c0a58ec-6d24-4037-973b-4450e76ce371","Type":"ContainerDied","Data":"b1d51d57af98fe7f586a0076f7a9c320074c38b8673bdd249e5c9c2cbf0d2197"} Nov 04 01:24:29.800181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.788369 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc"] Nov 04 01:24:29.800181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.789600 5051 cpu_manager.go:414] "containerMap: removing container" podUID="02116ac0-a24a-4c8f-a87c-e206df67e1ee" containerName="revoke-keys" Nov 04 01:24:29.800181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.789624 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02116ac0-a24a-4c8f-a87c-e206df67e1ee" containerName="revoke-keys" Nov 04 01:24:29.800181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.789801 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="02116ac0-a24a-4c8f-a87c-e206df67e1ee" containerName="revoke-keys" Nov 04 01:24:29.800181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.794512 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.806870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.805501 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7b10de73-gw-openshift-default-dockercfg-6wc47\"" Nov 04 01:24:29.831408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.831376 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc"] Nov 04 01:24:29.931307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931265 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b73b39-ac9b-4d1d-ae58-feb773882733-istiod-ca-cert\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.931489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931340 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-token\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.931489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931404 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-data\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.931595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931567 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-certs\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.931629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931612 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-envoy\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.931711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931698 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-credential-socket\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.931802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931786 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zlf\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-kube-api-access-x9zlf\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.931892 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931880 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-socket\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:29.931963 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:29.931951 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-podinfo\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.090338 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zlf\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-kube-api-access-x9zlf\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.090436 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-socket\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.090562 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-podinfo\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.090674 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b73b39-ac9b-4d1d-ae58-feb773882733-istiod-ca-cert\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.090750 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-token\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.090786 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-data\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.090996 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-certs\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.091047 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-envoy\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.091205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.091145 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-credential-socket\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.098149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.092341 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-credential-socket\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.098149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.092610 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b73b39-ac9b-4d1d-ae58-feb773882733-istiod-ca-cert\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.098149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.092995 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-data\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.098818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.098752 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-certs\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.099556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.099530 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-socket\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.168142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.153392 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-podinfo\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.168142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.153831 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-envoy\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.168142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.155021 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zlf\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-kube-api-access-x9zlf\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.168142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.166704 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-token\") pod \"e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.422199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.417470 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:30.835805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.835402 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:30.859986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.841643 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc"] Nov 04 01:24:30.899393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.897224 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:24:30.899393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.897350 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:24:30.899393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.897394 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:24:30.936145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.920892 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0a58ec-6d24-4037-973b-4450e76ce371-kube-api-access-sg4bc\" (UniqueName: \"kubernetes.io/projected/1c0a58ec-6d24-4037-973b-4450e76ce371-kube-api-access-sg4bc\") pod \"1c0a58ec-6d24-4037-973b-4450e76ce371\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " Nov 04 01:24:30.949190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.938281 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-openshift-ca\") pod \"1c0a58ec-6d24-4037-973b-4450e76ce371\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " Nov 04 01:24:30.949190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.938348 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-xks-ca\") pod \"1c0a58ec-6d24-4037-973b-4450e76ce371\" (UID: \"1c0a58ec-6d24-4037-973b-4450e76ce371\") " Nov 04 01:24:30.949190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.946534 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-openshift-ca" pod "1c0a58ec-6d24-4037-973b-4450e76ce371" (UID: "1c0a58ec-6d24-4037-973b-4450e76ce371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:30.949190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.946663 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-xks-ca" pod "1c0a58ec-6d24-4037-973b-4450e76ce371" (UID: "1c0a58ec-6d24-4037-973b-4450e76ce371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:30.973289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:30.972880 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0a58ec-6d24-4037-973b-4450e76ce371-kube-api-access-sg4bc" pod "1c0a58ec-6d24-4037-973b-4450e76ce371" (UID: "1c0a58ec-6d24-4037-973b-4450e76ce371"). InnerVolumeSpecName "kube-api-access-sg4bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:31.048445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.048407 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:31.048445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.048447 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0a58ec-6d24-4037-973b-4450e76ce371-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:31.048713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.048463 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4bc\" (UniqueName: \"kubernetes.io/projected/1c0a58ec-6d24-4037-973b-4450e76ce371-kube-api-access-sg4bc\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:31.547346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.547309 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" event={"ID":"10b73b39-ac9b-4d1d-ae58-feb773882733","Type":"ContainerStarted","Data":"5a02099e879a51bd63614a5f2dbd94491b669d05e408de5df7f8a9c331d25d6a"} Nov 04 01:24:31.547534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.547374 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" event={"ID":"10b73b39-ac9b-4d1d-ae58-feb773882733","Type":"ContainerStarted","Data":"19592f583a10359d3eae2834a5fc860140a5b6dc9d0d1023f692f4f504ff8d39"} Nov 04 01:24:31.548932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.548909 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" Nov 04 01:24:31.549056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.548957 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd" event={"ID":"1c0a58ec-6d24-4037-973b-4450e76ce371","Type":"ContainerDied","Data":"dc15b7ae5fbe7f6d72ba93d8221e2870357e2cf7c070fda16f5592c04a1c711e"} Nov 04 01:24:31.549056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.549005 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc15b7ae5fbe7f6d72ba93d8221e2870357e2cf7c070fda16f5592c04a1c711e" Nov 04 01:24:31.593472 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:31.593404 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" podStartSLOduration=2.5933848250000002 podStartE2EDuration="2.593384825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:31.585297689 +0000 UTC m=+1610.557046561" watchObservedRunningTime="2026-11-04 01:24:31.593384825 +0000 UTC m=+1610.565133699" Nov 04 01:24:32.419147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:32.419083 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:32.424752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:32.424725 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:32.562010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:32.561976 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:32.563076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:32.563052 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:24:34.153855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.153824 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t"] Nov 04 01:24:34.155470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.155445 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1c0a58ec-6d24-4037-973b-4450e76ce371" containerName="revoke-keys" Nov 04 01:24:34.155616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.155603 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0a58ec-6d24-4037-973b-4450e76ce371" containerName="revoke-keys" Nov 04 01:24:34.155961 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.155949 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0a58ec-6d24-4037-973b-4450e76ce371" containerName="revoke-keys" Nov 04 01:24:34.190953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.190917 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t"] Nov 04 01:24:34.195252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.190987 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.211418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.211379 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:24:34.215131 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.212256 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:24:34.220414 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:24:34.220379 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:24:34.260783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.260740 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:24:34.261106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.261082 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:24:34.271836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.271805 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b192653-e694-480f-b460-d3e77cd65942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.271981 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.271961 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b192653-e694-480f-b460-d3e77cd65942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.272195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.272173 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blndl\" (UniqueName: \"kubernetes.io/projected/8b192653-e694-480f-b460-d3e77cd65942-kube-api-access-blndl\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.373560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.373522 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blndl\" (UniqueName: \"kubernetes.io/projected/8b192653-e694-480f-b460-d3e77cd65942-kube-api-access-blndl\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.373752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.373733 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b192653-e694-480f-b460-d3e77cd65942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.373858 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.373842 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b192653-e694-480f-b460-d3e77cd65942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.374784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.374750 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b192653-e694-480f-b460-d3e77cd65942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.381521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.381495 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b192653-e694-480f-b460-d3e77cd65942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.388836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.388810 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blndl\" (UniqueName: \"kubernetes.io/projected/8b192653-e694-480f-b460-d3e77cd65942-kube-api-access-blndl\") pod \"maas-api-revoke-sub-4ec3e095-kn46t\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.524489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.524449 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:34.607307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.604990 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g"] Nov 04 01:24:34.611818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.611261 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.804439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.804391 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g"] Nov 04 01:24:34.805859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.805823 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.806013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.805931 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pngh\" (UniqueName: \"kubernetes.io/projected/6f658636-2519-4a3d-88d5-e1a307648713-kube-api-access-5pngh\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.806983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.806205 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.902624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.902590 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk"] Nov 04 01:24:34.908176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.908106 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.908375 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.908353 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.908472 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.908455 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pngh\" (UniqueName: \"kubernetes.io/projected/6f658636-2519-4a3d-88d5-e1a307648713-kube-api-access-5pngh\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.911338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.911302 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:34.913110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.913085 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.932542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.919380 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:34.932542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.922610 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk"] Nov 04 01:24:34.932542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.924598 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7b10de73-dockercfg-f9wdg\"" Nov 04 01:24:34.932542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.931106 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7b10de73\"" Nov 04 01:24:34.965314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:34.965251 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pngh\" (UniqueName: \"kubernetes.io/projected/6f658636-2519-4a3d-88d5-e1a307648713-kube-api-access-5pngh\") pod \"maas-api-revoke-sub-5868b6c3-bxt2g\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:35.009991 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.009951 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzcs8\" (UniqueName: \"kubernetes.io/projected/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-kube-api-access-jzcs8\") pod \"payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk\" (UID: \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:35.010214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.010054 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk\" (UID: \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:35.015483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.015451 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg"] Nov 04 01:24:35.021700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.021673 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.040009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.039934 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d7024f0c-openshift-default-dockercfg-bzddd\"" Nov 04 01:24:35.053619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.053580 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg"] Nov 04 01:24:35.092447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.091538 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.156809 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzcs8\" (UniqueName: \"kubernetes.io/projected/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-kube-api-access-jzcs8\") pod \"payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk\" (UID: \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.156882 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.156950 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86md\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-kube-api-access-h86md\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.156982 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.157015 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk\" (UID: \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.157093 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.157153 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.157217 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84934a5-2c51-47d8-a0d7-929640068888-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.157314 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84934a5-2c51-47d8-a0d7-929640068888-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.157356 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.157384 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.164384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.158857 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk\" (UID: \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:35.274178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.273804 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84934a5-2c51-47d8-a0d7-929640068888-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.274178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.273872 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.274178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.273901 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.274481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.274201 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.274481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.274271 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86md\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-kube-api-access-h86md\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.274481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.274301 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.274481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.274386 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.274481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.274421 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.274716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.274491 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84934a5-2c51-47d8-a0d7-929640068888-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.363274 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.347843 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-data\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.551315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.460871 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-certs\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.551581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.461657 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84934a5-2c51-47d8-a0d7-929640068888-istiod-ca-cert\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.551732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.463068 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-credential-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.551846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.514577 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-socket\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.551927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.537579 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-envoy\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.595219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.592901 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzcs8\" (UniqueName: \"kubernetes.io/projected/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-kube-api-access-jzcs8\") pod \"payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk\" (UID: \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\") " pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:35.597654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.597628 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86md\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-kube-api-access-h86md\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.598265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.598182 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84934a5-2c51-47d8-a0d7-929640068888-istio-podinfo\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.598785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.598760 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-istio-token\") pod \"e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.601233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.600921 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t"] Nov 04 01:24:35.623150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.623022 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" event={"ID":"8b192653-e694-480f-b460-d3e77cd65942","Type":"ContainerStarted","Data":"19095bec87bcd3955493f6fd1865c49d89c8a7765d82a0c13c0d8a1f4bfc7568"} Nov 04 01:24:35.642863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.642663 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:35.870700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.869342 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:35.999607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:35.999300 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g"] Nov 04 01:24:36.047918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.047878 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg"] Nov 04 01:24:36.048717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.048682 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:24:36.048802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.048762 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:24:36.048859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.048809 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:24:36.300063 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:24:36.299994 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56cb3c5_2ef2_4d62_8d57_c0bd60beabfc.slice/crio-21a2a1c001382bca4722d11f501002cafd163820279898c476e6dbf1c90ab909 WatchSource:0}: Error finding container 21a2a1c001382bca4722d11f501002cafd163820279898c476e6dbf1c90ab909: Status 404 returned error can't find the container with id 21a2a1c001382bca4722d11f501002cafd163820279898c476e6dbf1c90ab909 Nov 04 01:24:36.306691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.306664 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk"] Nov 04 01:24:36.634761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.634722 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" event={"ID":"6f658636-2519-4a3d-88d5-e1a307648713","Type":"ContainerStarted","Data":"982336d56e500582c764d3c3d02cff61201bb52d33948067de981ad2fc8b8265"} Nov 04 01:24:36.634952 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.634781 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" event={"ID":"6f658636-2519-4a3d-88d5-e1a307648713","Type":"ContainerStarted","Data":"033fbde234a28569fc854879d8b34449dd061681c7bc3f620bad8c5d03bb70d4"} Nov 04 01:24:36.636300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.636276 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" event={"ID":"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc","Type":"ContainerStarted","Data":"21a2a1c001382bca4722d11f501002cafd163820279898c476e6dbf1c90ab909"} Nov 04 01:24:36.638023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.637999 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" event={"ID":"c84934a5-2c51-47d8-a0d7-929640068888","Type":"ContainerStarted","Data":"9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192"} Nov 04 01:24:36.638143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.638046 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" event={"ID":"c84934a5-2c51-47d8-a0d7-929640068888","Type":"ContainerStarted","Data":"030d00bcc99757b7c2d2afe8608f6dfc42018d3cdc05108970871b0585ff4643"} Nov 04 01:24:36.639760 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.639732 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" event={"ID":"8b192653-e694-480f-b460-d3e77cd65942","Type":"ContainerStarted","Data":"fd9c3003ffcb4933cf275f9a8f7078de9be0d037e1394f62d82dbd30c27950fa"} Nov 04 01:24:36.643058 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.643008 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:36.659501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.659440 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" podStartSLOduration=2.6594232890000002 podStartE2EDuration="2.659423289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:36.657373037 +0000 UTC m=+1615.629121911" watchObservedRunningTime="2026-11-04 01:24:36.659423289 +0000 UTC m=+1615.631172154" Nov 04 01:24:36.698581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.698450 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" podStartSLOduration=2.698415051 podStartE2EDuration="2.698415051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:36.686389918 +0000 UTC m=+1615.658138802" watchObservedRunningTime="2026-11-04 01:24:36.698415051 +0000 UTC m=+1615.670163922" Nov 04 01:24:36.718619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:36.718567 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" podStartSLOduration=3.718552007 podStartE2EDuration="3.718552007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:36.705832225 +0000 UTC m=+1615.677581099" watchObservedRunningTime="2026-11-04 01:24:36.718552007 +0000 UTC m=+1615.690300915" Nov 04 01:24:37.649291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:37.649260 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:37.652595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:37.652561 5051 generic.go:356] "Generic (PLEG): container finished" podID="8b192653-e694-480f-b460-d3e77cd65942" containerID="fd9c3003ffcb4933cf275f9a8f7078de9be0d037e1394f62d82dbd30c27950fa" exitCode=0 Nov 04 01:24:37.652816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:37.652600 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" event={"ID":"8b192653-e694-480f-b460-d3e77cd65942","Type":"ContainerDied","Data":"fd9c3003ffcb4933cf275f9a8f7078de9be0d037e1394f62d82dbd30c27950fa"} Nov 04 01:24:37.654939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:37.654910 5051 generic.go:356] "Generic (PLEG): container finished" podID="6f658636-2519-4a3d-88d5-e1a307648713" containerID="982336d56e500582c764d3c3d02cff61201bb52d33948067de981ad2fc8b8265" exitCode=0 Nov 04 01:24:37.655048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:37.654988 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" event={"ID":"6f658636-2519-4a3d-88d5-e1a307648713","Type":"ContainerDied","Data":"982336d56e500582c764d3c3d02cff61201bb52d33948067de981ad2fc8b8265"} Nov 04 01:24:37.655443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:37.655426 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:37.656420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:37.656401 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:24:39.000311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.000274 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:39.056512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.053749 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:39.058754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.058720 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b192653-e694-480f-b460-d3e77cd65942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b192653-e694-480f-b460-d3e77cd65942-maas-api-openshift-ca\") pod \"8b192653-e694-480f-b460-d3e77cd65942\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " Nov 04 01:24:39.058878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.058848 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-xks-ca\") pod \"6f658636-2519-4a3d-88d5-e1a307648713\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " Nov 04 01:24:39.059051 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.059034 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b192653-e694-480f-b460-d3e77cd65942-kube-api-access-blndl\" (UniqueName: \"kubernetes.io/projected/8b192653-e694-480f-b460-d3e77cd65942-kube-api-access-blndl\") pod \"8b192653-e694-480f-b460-d3e77cd65942\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " Nov 04 01:24:39.059932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.059445 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-openshift-ca" pod "6f658636-2519-4a3d-88d5-e1a307648713" (UID: "6f658636-2519-4a3d-88d5-e1a307648713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:39.059932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.059468 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-openshift-ca\") pod \"6f658636-2519-4a3d-88d5-e1a307648713\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " Nov 04 01:24:39.059932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.059618 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f658636-2519-4a3d-88d5-e1a307648713-kube-api-access-5pngh\" (UniqueName: \"kubernetes.io/projected/6f658636-2519-4a3d-88d5-e1a307648713-kube-api-access-5pngh\") pod \"6f658636-2519-4a3d-88d5-e1a307648713\" (UID: \"6f658636-2519-4a3d-88d5-e1a307648713\") " Nov 04 01:24:39.059932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.059634 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b192653-e694-480f-b460-d3e77cd65942-maas-api-openshift-ca" pod "8b192653-e694-480f-b460-d3e77cd65942" (UID: "8b192653-e694-480f-b460-d3e77cd65942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:24:39.059932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.059739 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b192653-e694-480f-b460-d3e77cd65942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b192653-e694-480f-b460-d3e77cd65942-maas-api-xks-ca\") pod \"8b192653-e694-480f-b460-d3e77cd65942\" (UID: \"8b192653-e694-480f-b460-d3e77cd65942\") " Nov 04 01:24:39.060802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.060775 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:39.060912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.060807 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b192653-e694-480f-b460-d3e77cd65942-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:39.065078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.065048 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b192653-e694-480f-b460-d3e77cd65942-maas-api-xks-ca" pod "8b192653-e694-480f-b460-d3e77cd65942" (UID: "8b192653-e694-480f-b460-d3e77cd65942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:39.065654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.065619 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-xks-ca" pod "6f658636-2519-4a3d-88d5-e1a307648713" (UID: "6f658636-2519-4a3d-88d5-e1a307648713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:24:39.065740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.065664 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b192653-e694-480f-b460-d3e77cd65942-kube-api-access-blndl" pod "8b192653-e694-480f-b460-d3e77cd65942" (UID: "8b192653-e694-480f-b460-d3e77cd65942"). InnerVolumeSpecName "kube-api-access-blndl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:39.067878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.067835 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f658636-2519-4a3d-88d5-e1a307648713-kube-api-access-5pngh" pod "6f658636-2519-4a3d-88d5-e1a307648713" (UID: "6f658636-2519-4a3d-88d5-e1a307648713"). InnerVolumeSpecName "kube-api-access-5pngh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:24:39.163223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.163182 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f658636-2519-4a3d-88d5-e1a307648713-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:39.163223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.163222 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blndl\" (UniqueName: \"kubernetes.io/projected/8b192653-e694-480f-b460-d3e77cd65942-kube-api-access-blndl\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:39.163480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.163241 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pngh\" (UniqueName: \"kubernetes.io/projected/6f658636-2519-4a3d-88d5-e1a307648713-kube-api-access-5pngh\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:39.163480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:39.163255 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b192653-e694-480f-b460-d3e77cd65942-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:24:40.121159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:40.117176 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" event={"ID":"8b192653-e694-480f-b460-d3e77cd65942","Type":"ContainerDied","Data":"19095bec87bcd3955493f6fd1865c49d89c8a7765d82a0c13c0d8a1f4bfc7568"} Nov 04 01:24:40.121159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:40.117242 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19095bec87bcd3955493f6fd1865c49d89c8a7765d82a0c13c0d8a1f4bfc7568" Nov 04 01:24:40.121159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:40.117345 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t" Nov 04 01:24:40.128538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:40.128499 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" event={"ID":"6f658636-2519-4a3d-88d5-e1a307648713","Type":"ContainerDied","Data":"033fbde234a28569fc854879d8b34449dd061681c7bc3f620bad8c5d03bb70d4"} Nov 04 01:24:40.128688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:40.128566 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033fbde234a28569fc854879d8b34449dd061681c7bc3f620bad8c5d03bb70d4" Nov 04 01:24:40.128688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:40.128631 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g" Nov 04 01:24:43.232174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:43.231102 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" event={"ID":"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc","Type":"ContainerStarted","Data":"01f5c47c2b340ca4dcf8eb8ec450e0cd27ac0f011963aac094aeea8a3d14e680"} Nov 04 01:24:43.232174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:43.231226 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:43.256084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:43.256019 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" podStartSLOduration=3.187225225 podStartE2EDuration="9.256002448s" totalImagesPullingTime="6.068777223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:43.248470545 +0000 UTC m=+1622.220219419" watchObservedRunningTime="2026-11-04 01:24:43.256002448 +0000 UTC m=+1622.227751320" Nov 04 01:24:44.719152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:44.705212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:44.719152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:44.715257 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:44.880145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:44.876768 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.009151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.008211 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.009151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.008516 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.027144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.015190 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.045147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.037202 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.351204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.350414 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:45.351204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.350498 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.577147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.576258 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.725202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.712187 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.725202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.712756 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:24:45.725202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.723226 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:45.725202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.723815 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:24:45.828146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.810459 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:45.828146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.810535 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:45.881146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:45.868948 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:24:46.027153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.004870 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.086150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.011444 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:46.086150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.011494 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.086150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.028172 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:24:46.086150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.028241 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:24:46.086150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.019215 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.086150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.040939 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:24:46.086150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.041737 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.086150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.042047 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:24:46.164145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.139565 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:24:46.771206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.595481 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:46.771206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.599766 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:24:46.870626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.725689 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:46.894628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:46.726805 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.049054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:47.048949 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.050538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:47.050488 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:47.051065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:47.051029 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.068855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:47.068791 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:47.164147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:47.146421 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:47.210144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:47.206294 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.624145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:47.620349 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:47.931319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:47.931210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:48.015655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.015611 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:48.098670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.068178 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.099735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.095389 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:48.100421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.095425 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:48.126292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.096921 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:48.159366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.142253 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:48.159366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.142333 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.159987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.159947 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:48.292276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.292233 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:48.292564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.292533 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.659206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.637958 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:24:48.697198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.661237 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:48.697531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.697477 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:48.746470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:48.746410 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:49.056152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.055997 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.204084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.204021 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:49.332552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.332441 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:24:49.477147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.450485 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:49.477147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.461379 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:49.477147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.461435 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:49.477147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.461460 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:49.477147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.461503 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:49.477147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.464410 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.551250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.549710 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:49.551250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.549783 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.580350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.580310 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:49.580629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.580598 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.793153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.762361 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:49.996147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.993930 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj"] Nov 04 01:24:50.010163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.998318 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8b192653-e694-480f-b460-d3e77cd65942" containerName="revoke-keys" Nov 04 01:24:50.010163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.998352 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b192653-e694-480f-b460-d3e77cd65942" containerName="revoke-keys" Nov 04 01:24:50.010163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.998388 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6f658636-2519-4a3d-88d5-e1a307648713" containerName="revoke-keys" Nov 04 01:24:50.010163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.998397 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f658636-2519-4a3d-88d5-e1a307648713" containerName="revoke-keys" Nov 04 01:24:50.010163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.998699 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b192653-e694-480f-b460-d3e77cd65942" containerName="revoke-keys" Nov 04 01:24:50.010163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:49.998724 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f658636-2519-4a3d-88d5-e1a307648713" containerName="revoke-keys" Nov 04 01:24:50.058940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.058830 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.073143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.073080 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.085094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.084504 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:24:50.100317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.100284 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:24:50.105150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.101305 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:24:50.105370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.102113 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:24:50.223166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.213240 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.409239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.286236 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.409239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.286345 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.409239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.286378 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7ww\" (UniqueName: \"kubernetes.io/projected/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-kube-api-access-cx7ww\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.409239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.404592 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.409239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.404740 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.409239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.404773 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7ww\" (UniqueName: \"kubernetes.io/projected/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-kube-api-access-cx7ww\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.429145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.427364 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.533146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.467233 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.533146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.467638 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:50.533146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.467938 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:50.533146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.476291 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:50.533146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.476619 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:50.533146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.492832 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7ww\" (UniqueName: \"kubernetes.io/projected/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-kube-api-access-cx7ww\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.535645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.535592 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3cc47b3-p8ptj\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.615261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.615221 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj"] Nov 04 01:24:50.671211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.654029 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:50.765275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.764963 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:24:50.796508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:50.796374 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.153239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.145981 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.231481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.231434 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:51.392598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.392497 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.520145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.467880 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:51.520145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.468191 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.520145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.469022 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.520145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.479213 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.520145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.479628 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:51.542104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.540236 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:51.804321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:51.804266 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:52.112302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.112205 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:52.112559 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.112528 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:52.152306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.152232 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:52.239428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.231871 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:52.596146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.592598 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:53.010317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.613509 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:53.010317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.764423 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:53.010317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.764677 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:53.010317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.764904 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:53.010317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.799294 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:53.010317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.997304 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:53.010317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.821058 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:53.025632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:52.985898 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:53.025632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:53.011282 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:53.189567 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:53.170481 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:53.278883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:53.277174 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:53.382166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:53.382105 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:53.382452 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:53.382417 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:54.033194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.027243 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:54.033194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.027461 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:54.033194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.029351 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:54.033194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.029771 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:54.113322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.113273 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:54.253157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.136342 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:54.253157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.189595 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:54.570941 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.566951 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:54.666205 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:24:54.634638 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.418s" Nov 04 01:24:54.989596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:54.989564 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:24:55.010158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.009045 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:55.010158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.009222 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:55.010158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.009634 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:55.031157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.010534 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:55.050638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.046209 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:55.050638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.047231 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:55.050638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.047565 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:55.051319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.050986 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:55.112146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.077268 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj"] Nov 04 01:24:55.159245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.086641 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:55.159830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.159712 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:55.247346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.247237 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:55.284888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.284841 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:55.285316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.284790 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:55.602330 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.602227 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:24:55.742818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.742769 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:55.743040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.742845 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:55.875210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.872095 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5"] Nov 04 01:24:55.950149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.947244 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:55.982671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.982641 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5"] Nov 04 01:24:55.988143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.982906 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" event={"ID":"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3","Type":"ContainerStarted","Data":"b85c6671e0211d0d2c23870a3ed67717561d120004c62b0fab27b28c8988b1be"} Nov 04 01:24:55.988375 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:55.983583 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.026275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.026234 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:24:56.026555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.026520 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:56.054183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.050287 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:56.054183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.050738 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:24:56.054183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.052218 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:24:56.054183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.052529 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:56.096161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.092433 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.096161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.092712 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.096161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.093199 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbcrf\" (UniqueName: \"kubernetes.io/projected/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-kube-api-access-dbcrf\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.225145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.221446 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.225145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.221527 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.225145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.221663 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbcrf\" (UniqueName: \"kubernetes.io/projected/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-kube-api-access-dbcrf\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.225145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.223192 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.290693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.284838 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:56.303158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.294417 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbcrf\" (UniqueName: \"kubernetes.io/projected/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-kube-api-access-dbcrf\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.303158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.296715 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5403a6-bk8h5\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.328143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.323942 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:56.328143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.324522 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:56.379732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.379699 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:24:56.519233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.519199 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:24:56.650302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:56.636927 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:57.062073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.061978 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:57.100144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.064299 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:24:57.122144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.107070 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:57.163786 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.162893 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:57.379144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.324084 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:57.379144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.324874 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:57.424068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.398897 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:57.424068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.400177 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:57.653197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.641503 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:57.931573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:57.927711 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" podStartSLOduration=10.927689575 podStartE2EDuration="10.927689575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:24:57.775626567 +0000 UTC m=+1636.747375439" watchObservedRunningTime="2026-11-04 01:24:57.927689575 +0000 UTC m=+1636.899438447" Nov 04 01:24:58.111310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.107023 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:58.111310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.107581 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:58.111310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.107988 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:58.433204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.327295 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:24:58.433204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.327671 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:24:58.433204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.425552 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:24:58.494305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.433699 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:24:58.494305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.434380 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:24:58.714201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.618857 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:24:58.714201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.675838 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:24:58.714201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.682627 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:24:59.012195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:58.991246 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:24:59.024666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.024632 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:24:59.036621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.036419 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:24:59.040265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.037841 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:24:59.043136 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.043099 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:24:59.043759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.043718 5051 generic.go:356] "Generic (PLEG): container finished" podID="a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3" containerID="a0d3595ba13e6d64fa596be1e9b7a18a79b1043f88d84a6f87acbdecae4c915a" exitCode=0 Nov 04 01:24:59.047752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.045142 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" event={"ID":"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3","Type":"ContainerDied","Data":"a0d3595ba13e6d64fa596be1e9b7a18a79b1043f88d84a6f87acbdecae4c915a"} Nov 04 01:24:59.047752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.045984 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:24:59.047752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.047059 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:24:59.095168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:24:59.094304 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5"] Nov 04 01:24:59.100742 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:24:59.099841 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14f96d4_9de5_4bfc_beef_9e7f76b7b4d3.slice/crio-a0d3595ba13e6d64fa596be1e9b7a18a79b1043f88d84a6f87acbdecae4c915a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14f96d4_9de5_4bfc_beef_9e7f76b7b4d3.slice/crio-conmon-a0d3595ba13e6d64fa596be1e9b7a18a79b1043f88d84a6f87acbdecae4c915a.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:25:00.152794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.152751 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" event={"ID":"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a","Type":"ContainerStarted","Data":"747858deade6e3a7f8a24e63e098427d810643138aaf11cd755f8d07ec13cc51"} Nov 04 01:25:00.153341 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.152811 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" event={"ID":"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a","Type":"ContainerStarted","Data":"c7e0cbf7a3a0e277386089001f49547bddddf10ab2f38562d5da1ac139f71eb5"} Nov 04 01:25:00.178021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.177974 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" podStartSLOduration=6.177958793 podStartE2EDuration="6.177958793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:00.172218131 +0000 UTC m=+1639.143967004" watchObservedRunningTime="2026-11-04 01:25:00.177958793 +0000 UTC m=+1639.149707663" Nov 04 01:25:00.347585 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.347560 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:25:00.451607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.451516 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-xks-ca\") pod \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " Nov 04 01:25:00.451768 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.451730 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-openshift-ca\") pod \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " Nov 04 01:25:00.451854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.451841 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-kube-api-access-cx7ww\" (UniqueName: \"kubernetes.io/projected/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-kube-api-access-cx7ww\") pod \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\" (UID: \"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3\") " Nov 04 01:25:00.452092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.452068 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-openshift-ca" pod "a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3" (UID: "a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:00.452705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.452682 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:00.453896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.453866 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-xks-ca" pod "a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3" (UID: "a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:00.454010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.453947 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-kube-api-access-cx7ww" pod "a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3" (UID: "a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3"). InnerVolumeSpecName "kube-api-access-cx7ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:00.553997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.553964 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7ww\" (UniqueName: \"kubernetes.io/projected/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-kube-api-access-cx7ww\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:00.553997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:00.553992 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:01.039641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.039589 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6"] Nov 04 01:25:01.046363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.046331 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd855fa7-bcrc6"] Nov 04 01:25:01.055477 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.053667 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb"] Nov 04 01:25:01.063500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.063471 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz"] Nov 04 01:25:01.072223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.072194 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e91b975-dhblb"] Nov 04 01:25:01.078797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.078769 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b495d9-5wrrz"] Nov 04 01:25:01.165832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.165799 5051 generic.go:356] "Generic (PLEG): container finished" podID="220ad717-fb38-4ec1-aaa3-6dc341f1fa3a" containerID="747858deade6e3a7f8a24e63e098427d810643138aaf11cd755f8d07ec13cc51" exitCode=0 Nov 04 01:25:01.166315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.165887 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" event={"ID":"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a","Type":"ContainerDied","Data":"747858deade6e3a7f8a24e63e098427d810643138aaf11cd755f8d07ec13cc51"} Nov 04 01:25:01.167543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.167523 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" Nov 04 01:25:01.167654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.167548 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj" event={"ID":"a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3","Type":"ContainerDied","Data":"b85c6671e0211d0d2c23870a3ed67717561d120004c62b0fab27b28c8988b1be"} Nov 04 01:25:01.167654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.167593 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b85c6671e0211d0d2c23870a3ed67717561d120004c62b0fab27b28c8988b1be" Nov 04 01:25:01.190291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.190269 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054f3b94-1e68-49d0-b304-58c17757c665" path="/var/lib/kubelet/pods/054f3b94-1e68-49d0-b304-58c17757c665/volumes" Nov 04 01:25:01.191348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.191308 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee" path="/var/lib/kubelet/pods/10c314ef-9509-4dd6-8e3d-ceedf1dcf5ee/volumes" Nov 04 01:25:01.193494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:01.193468 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27c1c45-93f0-4eb5-95a3-890e4549f600" path="/var/lib/kubelet/pods/f27c1c45-93f0-4eb5-95a3-890e4549f600/volumes" Nov 04 01:25:02.036280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.036248 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87"] Nov 04 01:25:02.042993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.042960 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq"] Nov 04 01:25:02.049569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.049538 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-522fee64-cgz87"] Nov 04 01:25:02.056432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.056401 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9d0485d-dwtlq"] Nov 04 01:25:02.404194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.404169 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:25:02.529105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.520866 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-kube-api-access-dbcrf\" (UniqueName: \"kubernetes.io/projected/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-kube-api-access-dbcrf\") pod \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " Nov 04 01:25:02.529105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.521219 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-xks-ca\") pod \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " Nov 04 01:25:02.529105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.521333 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-openshift-ca\") pod \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\" (UID: \"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a\") " Nov 04 01:25:02.552687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.548596 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-openshift-ca" pod "220ad717-fb38-4ec1-aaa3-6dc341f1fa3a" (UID: "220ad717-fb38-4ec1-aaa3-6dc341f1fa3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:02.569677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.566579 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-kube-api-access-dbcrf" pod "220ad717-fb38-4ec1-aaa3-6dc341f1fa3a" (UID: "220ad717-fb38-4ec1-aaa3-6dc341f1fa3a"). InnerVolumeSpecName "kube-api-access-dbcrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:02.569677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.569333 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-xks-ca" pod "220ad717-fb38-4ec1-aaa3-6dc341f1fa3a" (UID: "220ad717-fb38-4ec1-aaa3-6dc341f1fa3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:02.623505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.623446 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbcrf\" (UniqueName: \"kubernetes.io/projected/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-kube-api-access-dbcrf\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:02.623505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.623474 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:02.623505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:02.623491 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:03.192693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.192665 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e9077a-c19b-407d-9b95-c8b6a9bfa2b7" path="/var/lib/kubelet/pods/30e9077a-c19b-407d-9b95-c8b6a9bfa2b7/volumes" Nov 04 01:25:03.193334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.193308 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5211f4db-a6b4-4e7e-9447-010f5b7282d7" path="/var/lib/kubelet/pods/5211f4db-a6b4-4e7e-9447-010f5b7282d7/volumes" Nov 04 01:25:03.197852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.197824 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" Nov 04 01:25:03.198035 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.197817 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5" event={"ID":"220ad717-fb38-4ec1-aaa3-6dc341f1fa3a","Type":"ContainerDied","Data":"c7e0cbf7a3a0e277386089001f49547bddddf10ab2f38562d5da1ac139f71eb5"} Nov 04 01:25:03.198035 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.197977 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e0cbf7a3a0e277386089001f49547bddddf10ab2f38562d5da1ac139f71eb5" Nov 04 01:25:03.533993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.533953 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8"] Nov 04 01:25:03.535820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.535792 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3" containerName="revoke-keys" Nov 04 01:25:03.535820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.535817 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3" containerName="revoke-keys" Nov 04 01:25:03.536010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.535833 5051 cpu_manager.go:414] "containerMap: removing container" podUID="220ad717-fb38-4ec1-aaa3-6dc341f1fa3a" containerName="revoke-keys" Nov 04 01:25:03.536010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.535840 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220ad717-fb38-4ec1-aaa3-6dc341f1fa3a" containerName="revoke-keys" Nov 04 01:25:03.536166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.536114 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3" containerName="revoke-keys" Nov 04 01:25:03.536249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.536172 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="220ad717-fb38-4ec1-aaa3-6dc341f1fa3a" containerName="revoke-keys" Nov 04 01:25:03.553138 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.553091 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.556596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.556564 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8"] Nov 04 01:25:03.559949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.559843 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:25:03.559949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.559890 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:25:03.559949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.559916 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:25:03.560781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.560762 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:25:03.646576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.646541 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.646832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.646814 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.646892 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.646848 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrws2\" (UniqueName: \"kubernetes.io/projected/565a90b8-a794-4fad-9033-b68a5c8788ee-kube-api-access-jrws2\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.765144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.756691 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.765144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.756792 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrws2\" (UniqueName: \"kubernetes.io/projected/565a90b8-a794-4fad-9033-b68a5c8788ee-kube-api-access-jrws2\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.765144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.756976 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.765144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.759064 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.779138 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.775673 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.837186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.837058 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrws2\" (UniqueName: \"kubernetes.io/projected/565a90b8-a794-4fad-9033-b68a5c8788ee-kube-api-access-jrws2\") pod \"maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.856142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.856008 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24"] Nov 04 01:25:03.866761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.862948 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:03.868171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.867836 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24"] Nov 04 01:25:03.886982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.878438 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:03.962289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.961095 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4zb\" (UniqueName: \"kubernetes.io/projected/ac998ad5-d354-4564-86f9-f53b3cdd068f-kube-api-access-cs4zb\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:03.962289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.961308 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:03.962289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:03.961417 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:04.063735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.063699 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:04.064383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.064354 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:04.065301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.065137 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:04.065577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.065558 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4zb\" (UniqueName: \"kubernetes.io/projected/ac998ad5-d354-4564-86f9-f53b3cdd068f-kube-api-access-cs4zb\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:04.068546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.068517 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:04.082168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.076543 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4zb\" (UniqueName: \"kubernetes.io/projected/ac998ad5-d354-4564-86f9-f53b3cdd068f-kube-api-access-cs4zb\") pod \"maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:04.115423 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:04.115356 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:25:04.184710 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:25:04.184683 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565a90b8_a794_4fad_9033_b68a5c8788ee.slice/crio-478eceb32a6640caba94c24c6ad291978986d6cce0e6025c24595eab76d9ac2f WatchSource:0}: Error finding container 478eceb32a6640caba94c24c6ad291978986d6cce0e6025c24595eab76d9ac2f: Status 404 returned error can't find the container with id 478eceb32a6640caba94c24c6ad291978986d6cce0e6025c24595eab76d9ac2f Nov 04 01:25:04.189654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.189633 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:04.189965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.189941 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8"] Nov 04 01:25:04.209368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.209345 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" event={"ID":"565a90b8-a794-4fad-9033-b68a5c8788ee","Type":"ContainerStarted","Data":"478eceb32a6640caba94c24c6ad291978986d6cce0e6025c24595eab76d9ac2f"} Nov 04 01:25:04.487292 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:25:04.487262 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac998ad5_d354_4564_86f9_f53b3cdd068f.slice/crio-2f7cf652a63ae13be5ac19f426887432bf818c8aedd66026d40088e2c08984ca WatchSource:0}: Error finding container 2f7cf652a63ae13be5ac19f426887432bf818c8aedd66026d40088e2c08984ca: Status 404 returned error can't find the container with id 2f7cf652a63ae13be5ac19f426887432bf818c8aedd66026d40088e2c08984ca Nov 04 01:25:04.490363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:04.490063 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24"] Nov 04 01:25:05.221275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:05.221180 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" event={"ID":"565a90b8-a794-4fad-9033-b68a5c8788ee","Type":"ContainerStarted","Data":"5e09dd83f2cc613c0121b10a01c8dd41407cdded64d054e1a367123fcacb70af"} Nov 04 01:25:05.223209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:05.223179 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" event={"ID":"ac998ad5-d354-4564-86f9-f53b3cdd068f","Type":"ContainerStarted","Data":"ecfad41ddb655cd9bc43c6c51e9e7d2f5fc0b76f3259094afd5a304f4f87185e"} Nov 04 01:25:05.223344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:05.223230 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" event={"ID":"ac998ad5-d354-4564-86f9-f53b3cdd068f","Type":"ContainerStarted","Data":"2f7cf652a63ae13be5ac19f426887432bf818c8aedd66026d40088e2c08984ca"} Nov 04 01:25:05.245195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:05.245103 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" podStartSLOduration=2.245086871 podStartE2EDuration="2.245086871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:05.243597072 +0000 UTC m=+1644.215345943" watchObservedRunningTime="2026-11-04 01:25:05.245086871 +0000 UTC m=+1644.216835744" Nov 04 01:25:05.275263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:05.275202 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" podStartSLOduration=2.275183542 podStartE2EDuration="2.275183542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:05.265977395 +0000 UTC m=+1644.237726266" watchObservedRunningTime="2026-11-04 01:25:05.275183542 +0000 UTC m=+1644.246932414" Nov 04 01:25:06.239563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:06.239522 5051 generic.go:356] "Generic (PLEG): container finished" podID="565a90b8-a794-4fad-9033-b68a5c8788ee" containerID="5e09dd83f2cc613c0121b10a01c8dd41407cdded64d054e1a367123fcacb70af" exitCode=0 Nov 04 01:25:06.240065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:06.239601 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" event={"ID":"565a90b8-a794-4fad-9033-b68a5c8788ee","Type":"ContainerDied","Data":"5e09dd83f2cc613c0121b10a01c8dd41407cdded64d054e1a367123fcacb70af"} Nov 04 01:25:06.241548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:06.241526 5051 generic.go:356] "Generic (PLEG): container finished" podID="ac998ad5-d354-4564-86f9-f53b3cdd068f" containerID="ecfad41ddb655cd9bc43c6c51e9e7d2f5fc0b76f3259094afd5a304f4f87185e" exitCode=0 Nov 04 01:25:06.241626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:06.241590 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" event={"ID":"ac998ad5-d354-4564-86f9-f53b3cdd068f","Type":"ContainerDied","Data":"ecfad41ddb655cd9bc43c6c51e9e7d2f5fc0b76f3259094afd5a304f4f87185e"} Nov 04 01:25:07.501718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.500840 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:07.506192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.506165 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:07.540950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.540658 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac998ad5-d354-4564-86f9-f53b3cdd068f-kube-api-access-cs4zb\" (UniqueName: \"kubernetes.io/projected/ac998ad5-d354-4564-86f9-f53b3cdd068f-kube-api-access-cs4zb\") pod \"ac998ad5-d354-4564-86f9-f53b3cdd068f\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " Nov 04 01:25:07.540950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.540731 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-openshift-ca\") pod \"565a90b8-a794-4fad-9033-b68a5c8788ee\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " Nov 04 01:25:07.541940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.541525 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-xks-ca\") pod \"565a90b8-a794-4fad-9033-b68a5c8788ee\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " Nov 04 01:25:07.541940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.541572 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-xks-ca\") pod \"ac998ad5-d354-4564-86f9-f53b3cdd068f\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " Nov 04 01:25:07.541940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.541628 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-openshift-ca\") pod \"ac998ad5-d354-4564-86f9-f53b3cdd068f\" (UID: \"ac998ad5-d354-4564-86f9-f53b3cdd068f\") " Nov 04 01:25:07.541940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.541762 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/565a90b8-a794-4fad-9033-b68a5c8788ee-kube-api-access-jrws2\" (UniqueName: \"kubernetes.io/projected/565a90b8-a794-4fad-9033-b68a5c8788ee-kube-api-access-jrws2\") pod \"565a90b8-a794-4fad-9033-b68a5c8788ee\" (UID: \"565a90b8-a794-4fad-9033-b68a5c8788ee\") " Nov 04 01:25:07.545102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.545068 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-openshift-ca" pod "ac998ad5-d354-4564-86f9-f53b3cdd068f" (UID: "ac998ad5-d354-4564-86f9-f53b3cdd068f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:07.545542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.545513 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-openshift-ca" pod "565a90b8-a794-4fad-9033-b68a5c8788ee" (UID: "565a90b8-a794-4fad-9033-b68a5c8788ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:07.546389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.546368 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565a90b8-a794-4fad-9033-b68a5c8788ee-kube-api-access-jrws2" pod "565a90b8-a794-4fad-9033-b68a5c8788ee" (UID: "565a90b8-a794-4fad-9033-b68a5c8788ee"). InnerVolumeSpecName "kube-api-access-jrws2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:07.546516 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.546416 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-xks-ca" pod "565a90b8-a794-4fad-9033-b68a5c8788ee" (UID: "565a90b8-a794-4fad-9033-b68a5c8788ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:07.546615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.546590 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac998ad5-d354-4564-86f9-f53b3cdd068f-kube-api-access-cs4zb" pod "ac998ad5-d354-4564-86f9-f53b3cdd068f" (UID: "ac998ad5-d354-4564-86f9-f53b3cdd068f"). InnerVolumeSpecName "kube-api-access-cs4zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:07.549288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.549261 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-xks-ca" pod "ac998ad5-d354-4564-86f9-f53b3cdd068f" (UID: "ac998ad5-d354-4564-86f9-f53b3cdd068f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:07.644557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.644508 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:07.644557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.644556 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/565a90b8-a794-4fad-9033-b68a5c8788ee-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:07.644557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.644571 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:07.644789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.644584 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac998ad5-d354-4564-86f9-f53b3cdd068f-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:07.644789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.644597 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrws2\" (UniqueName: \"kubernetes.io/projected/565a90b8-a794-4fad-9033-b68a5c8788ee-kube-api-access-jrws2\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:07.644789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:07.644609 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4zb\" (UniqueName: \"kubernetes.io/projected/ac998ad5-d354-4564-86f9-f53b3cdd068f-kube-api-access-cs4zb\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:08.262901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:08.262866 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" Nov 04 01:25:08.262901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:08.262883 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8" event={"ID":"565a90b8-a794-4fad-9033-b68a5c8788ee","Type":"ContainerDied","Data":"478eceb32a6640caba94c24c6ad291978986d6cce0e6025c24595eab76d9ac2f"} Nov 04 01:25:08.263217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:08.262930 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="478eceb32a6640caba94c24c6ad291978986d6cce0e6025c24595eab76d9ac2f" Nov 04 01:25:08.264518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:08.264495 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" event={"ID":"ac998ad5-d354-4564-86f9-f53b3cdd068f","Type":"ContainerDied","Data":"2f7cf652a63ae13be5ac19f426887432bf818c8aedd66026d40088e2c08984ca"} Nov 04 01:25:08.264518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:08.264514 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24" Nov 04 01:25:08.264688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:08.264533 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f7cf652a63ae13be5ac19f426887432bf818c8aedd66026d40088e2c08984ca" Nov 04 01:25:10.034040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:10.034002 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k"] Nov 04 01:25:10.040055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:10.040012 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd365704-mbf6k"] Nov 04 01:25:11.081305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:11.081191 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5"] Nov 04 01:25:11.144180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:11.135596 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d113800-qlsp5"] Nov 04 01:25:11.196275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:11.196241 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7158263f-6f8d-4f6c-835b-36899206e24d" path="/var/lib/kubelet/pods/7158263f-6f8d-4f6c-835b-36899206e24d/volumes" Nov 04 01:25:11.196892 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:11.196874 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97950fd-de4b-4166-8555-a4eba0ebaf0d" path="/var/lib/kubelet/pods/a97950fd-de4b-4166-8555-a4eba0ebaf0d/volumes" Nov 04 01:25:13.967141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:13.942403 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:14.135044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:14.134966 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:14.604155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:14.581614 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk"] Nov 04 01:25:15.085071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.080645 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.133623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.133579 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.140260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.140214 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:15.166307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.165419 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:15.206146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.165481 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:15.207073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.165511 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.213951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.180101 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:25:15.215553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.214673 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:15.939555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.937961 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:25:15.940256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:15.940201 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.083254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.083203 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.084284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.084257 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:25:16.145639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.145253 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:25:16.149646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.149610 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:25:16.173202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.172423 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:16.182817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.174104 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:16.182817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.175542 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:25:16.278169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.268212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:16.278169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.268793 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:25:16.278169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.275495 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.326688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.311822 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:25:16.328335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.328297 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:16.329261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:16.329242 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:25:17.037145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.034069 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.073144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.045268 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:25:17.147525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.147476 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:25:17.160308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.160253 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.318399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.318286 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:25:17.501570 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.501522 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Nov 04 01:25:17.688887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.688769 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:17.750113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.750075 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:17.750447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.750413 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.782286 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.782225 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:17.888772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.888725 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:17.988163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.988107 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:18.019291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:17.990734 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" podUID="f80b3f88-24db-4e50-aecc-c1c19664e0ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 04 01:25:18.020717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:18.020666 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.129876 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:18.129828 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.204649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:18.204605 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.337621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:18.337509 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:18.590205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:18.577328 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.772690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:18.767570 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:25:18.789624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:18.789571 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:18.837957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:18.802465 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.015397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.015353 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:19.070147 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:19.054881 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.848s" Nov 04 01:25:19.132483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.132437 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:25:19.234760 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.234713 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.336309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.336202 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:25:19.361253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.361198 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.361451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.361280 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:19.590223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.582539 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt"] Nov 04 01:25:19.590459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.582846 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:25:19.606391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.606345 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" podUID="b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" containerName="payload-processing" containerID="cri-o://01f5c47c2b340ca4dcf8eb8ec450e0cd27ac0f011963aac094aeea8a3d14e680" gracePeriod=25 Nov 04 01:25:19.839146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.800181 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.839146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.800681 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 04 01:25:19.839146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.801527 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9207e79f-zgbtt"] Nov 04 01:25:19.871333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.840151 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:19.871333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:19.840741 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:20.055167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.034439 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.184148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.135301 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.285159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.260903 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.368496 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.365554 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.612289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.612242 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:25:20.645316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.645260 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Nov 04 01:25:20.824143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.800363 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.852148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.850029 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:20.852546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:20.852460 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:21.074176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.062094 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:21.080384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.080345 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2"] Nov 04 01:25:21.135218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.104282 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ac998ad5-d354-4564-86f9-f53b3cdd068f" containerName="revoke-keys" Nov 04 01:25:21.135471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.135440 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac998ad5-d354-4564-86f9-f53b3cdd068f" containerName="revoke-keys" Nov 04 01:25:21.141408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.141374 5051 cpu_manager.go:414] "containerMap: removing container" podUID="565a90b8-a794-4fad-9033-b68a5c8788ee" containerName="revoke-keys" Nov 04 01:25:21.148201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.148158 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="565a90b8-a794-4fad-9033-b68a5c8788ee" containerName="revoke-keys" Nov 04 01:25:21.163151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.148712 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac998ad5-d354-4564-86f9-f53b3cdd068f" containerName="revoke-keys" Nov 04 01:25:21.182166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.163373 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="565a90b8-a794-4fad-9033-b68a5c8788ee" containerName="revoke-keys" Nov 04 01:25:21.216183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.166765 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:21.399215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.398214 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:21.399215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.398489 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:21.424158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.409302 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:21.557218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.506092 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:21.557218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.506172 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:21.557218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.506300 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng45v\" (UniqueName: \"kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:21.653494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.605014 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:21.753471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.621289 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.623944 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng45v\" (UniqueName: \"kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.764337 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.764420 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:21.741672 5051 projected.go:291] Couldn't get configMap odh-ai-gateway-infra/kube-root-ca.crt: object "odh-ai-gateway-infra"/"kube-root-ca.crt" not registered Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:21.764732 5051 projected.go:291] Couldn't get configMap odh-ai-gateway-infra/openshift-service-ca.crt: object "odh-ai-gateway-infra"/"openshift-service-ca.crt" not registered Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:21.764747 5051 projected.go:196] Error preparing data for projected volume kube-api-access-ng45v for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2: [object "odh-ai-gateway-infra"/"kube-root-ca.crt" not registered, object "odh-ai-gateway-infra"/"openshift-service-ca.crt" not registered] Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:21.764809 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v podName:82e10c2c-739e-441c-9a7a-9385e2f0d26f nodeName:}" failed. No retries permitted until 2026-11-04 01:25:22.264789018 +0000 UTC m=+1661.236537888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ng45v" (UniqueName: "kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v") pod "maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" (UID: "82e10c2c-739e-441c-9a7a-9385e2f0d26f") : [object "odh-ai-gateway-infra"/"kube-root-ca.crt" not registered, object "odh-ai-gateway-infra"/"openshift-service-ca.crt" not registered] Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:21.764701 5051 configmap.go:193] Couldn't get configMap odh-ai-gateway-infra/openshift-service-ca.crt: object "odh-ai-gateway-infra"/"openshift-service-ca.crt" not registered Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:21.764842 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca podName:82e10c2c-739e-441c-9a7a-9385e2f0d26f nodeName:}" failed. No retries permitted until 2026-11-04 01:25:22.264834309 +0000 UTC m=+1661.236583172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-openshift-ca" (UniqueName: "kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca") pod "maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" (UID: "82e10c2c-739e-441c-9a7a-9385e2f0d26f") : object "odh-ai-gateway-infra"/"openshift-service-ca.crt" not registered Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:21.764916 5051 secret.go:189] Couldn't get secret odh-ai-gateway-infra/opendatahub-ca: object "odh-ai-gateway-infra"/"opendatahub-ca" not registered Nov 04 01:25:21.802990 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:21.764952 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca podName:82e10c2c-739e-441c-9a7a-9385e2f0d26f nodeName:}" failed. No retries permitted until 2026-11-04 01:25:22.264940405 +0000 UTC m=+1661.236689257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-xks-ca" (UniqueName: "kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca") pod "maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" (UID: "82e10c2c-739e-441c-9a7a-9385e2f0d26f") : object "odh-ai-gateway-infra"/"opendatahub-ca" not registered Nov 04 01:25:21.865163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.843250 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:21.905614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.880424 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:21.905889 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:21.891799 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.084161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.042653 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2"] Nov 04 01:25:22.141379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.141329 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:22.277145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.250513 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:22.406145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.393610 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:22.573213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.555864 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:25:22.573213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.556597 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:25:22.573213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.571940 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:25:22.620358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.614705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:22.727148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.704318 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:23.202147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.201493 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng45v\" (UniqueName: \"kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:23.331212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.254606 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.331212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.143956 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:23.331212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.180045 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:23.331212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.269226 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.331212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.200504 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:25:23.331212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.652209 5051 generic.go:356] "Generic (PLEG): container finished" podID="b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" containerID="01f5c47c2b340ca4dcf8eb8ec450e0cd27ac0f011963aac094aeea8a3d14e680" exitCode=0 Nov 04 01:25:23.405088 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.200574 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:23.528146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.523558 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.569147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.303513 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.622191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.200273 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:23.622191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.597691 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:23.826202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:22.616661 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:25:23.896144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.892917 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb591b4a-c0f6-448b-974d-023ca9620ecb" path="/var/lib/kubelet/pods/fb591b4a-c0f6-448b-974d-023ca9620ecb/volumes" Nov 04 01:25:24.172353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.984245 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:25:24.172353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:23.993527 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.176912 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:24.176870 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.134s" Nov 04 01:25:24.177206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.177168 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" event={"ID":"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc","Type":"ContainerDied","Data":"01f5c47c2b340ca4dcf8eb8ec450e0cd27ac0f011963aac094aeea8a3d14e680"} Nov 04 01:25:24.236151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.218162 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:25:24.306309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.306259 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.568012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.329218 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.568012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.329286 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:24.568012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.397416 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Nov 04 01:25:24.568012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.533678 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.568012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.533866 5051 trace.go:236] Trace[2050784724]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" (04-Nov-2026 01:25:23.182) (total time: 1351ms): Nov 04 01:25:24.568012 ip-10-0-137-48 kubenswrapper[5051]: Trace[2050784724]: [1.35172372s] [1.35172372s] END Nov 04 01:25:24.699142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.679525 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng45v\" (UniqueName: \"kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v\") pod \"maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:24.699142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.679720 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:24.699142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.680179 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:25:24.699142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.680293 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:25:24.699142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.680444 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:24.739591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.739558 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:24.894654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:24.894595 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" podUID="f80b3f88-24db-4e50-aecc-c1c19664e0ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:25.009829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.009771 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:25.054523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.054487 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:25.430775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.429309 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" podUID="b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.142:50052\" within 1s: context deadline exceeded" Nov 04 01:25:25.430775 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:25:25.429518 5051 logging.go:55] [core] [Channel #32 SubChannel #33]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.142:50052", ServerName: "10.132.0.142:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.142:50052: operation was canceled" Nov 04 01:25:25.471684 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:25.471646 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.294s" Nov 04 01:25:25.472179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.472100 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:25:25.565866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.565828 5051 patch_prober.go:29] interesting pod/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:25.566185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.566151 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:25.592763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.585375 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:25.592763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.585755 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:25.592763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.586019 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:25.629958 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.629916 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:25:25.630214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.629988 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8"] Nov 04 01:25:25.630214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.630032 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:25:25.630214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.630071 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:25:25.632508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.630114 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:25:25.681386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.679204 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d92c65cb-xgmk8"] Nov 04 01:25:25.709374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.709091 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:25.709374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.709191 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:25:25.709678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.709449 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:25.709678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.709516 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:25:25.891381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:25.857455 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:26.348819 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.334861 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:26.348819 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.334939 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:26.386757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.349299 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24"] Nov 04 01:25:26.438370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.437257 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:25:26.762937 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.762563 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:25:26.794581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.763877 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7b10de73-l4x24"] Nov 04 01:25:26.794581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.768016 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:25:26.794581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.792744 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:26.794581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.792800 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:26.945280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.945237 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:25:26.954140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.946404 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:27.011183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.981100 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:27.011183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.981779 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:27.011183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:26.983783 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:27.216239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.195282 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Nov 04 01:25:27.469660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.469565 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:27.652599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.650454 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="565a90b8-a794-4fad-9033-b68a5c8788ee" path="/var/lib/kubelet/pods/565a90b8-a794-4fad-9033-b68a5c8788ee/volumes" Nov 04 01:25:27.661280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.661245 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac998ad5-d354-4564-86f9-f53b3cdd068f" path="/var/lib/kubelet/pods/ac998ad5-d354-4564-86f9-f53b3cdd068f/volumes" Nov 04 01:25:27.663620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.662072 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc"] Nov 04 01:25:27.713374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.713338 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:25:27.713850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.713817 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" podUID="10b73b39-ac9b-4d1d-ae58-feb773882733" containerName="istio-proxy" containerID="cri-o://5a02099e879a51bd63614a5f2dbd94491b669d05e408de5df7f8a9c331d25d6a" gracePeriod=30 Nov 04 01:25:27.759377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.759344 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:25:27.812229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.768035 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:27.812229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.771672 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:27.812229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.785045 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf"] Nov 04 01:25:27.896300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.896267 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" containerName="payload-processing" Nov 04 01:25:27.896552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.896532 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" containerName="payload-processing" Nov 04 01:25:27.912144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.897009 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" containerName="payload-processing" Nov 04 01:25:28.586141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.579234 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:28.586141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.985206 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:28.586141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.985240 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:25:28.586141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.985265 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:28.587702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.011098 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-plugins-config-volume" pod "b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" (UID: "b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:28.587907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.062852 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:28.588588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.064191 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:28.623151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.623096 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:28.629739 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:27.980967 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-plugins-config-volume\") pod \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\" (UID: \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\") " Nov 04 01:25:28.657608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.657575 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-kube-api-access-jzcs8\" (UniqueName: \"kubernetes.io/projected/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-kube-api-access-jzcs8\") pod \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\" (UID: \"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc\") " Nov 04 01:25:28.799351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.799303 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:28.835412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.835376 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:28.860522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.827889 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:25:28.867797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.867762 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" Nov 04 01:25:29.026306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.970989 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk" event={"ID":"b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc","Type":"ContainerDied","Data":"21a2a1c001382bca4722d11f501002cafd163820279898c476e6dbf1c90ab909"} Nov 04 01:25:29.026306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.015002 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf"] Nov 04 01:25:29.026306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.021613 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-kube-api-access-jzcs8" pod "b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" (UID: "b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc"). InnerVolumeSpecName "kube-api-access-jzcs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:29.052377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.041791 5051 scope.go:122] "RemoveContainer" containerID="01f5c47c2b340ca4dcf8eb8ec450e0cd27ac0f011963aac094aeea8a3d14e680" Nov 04 01:25:29.052377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:28.971100 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.163552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.161794 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv"] Nov 04 01:25:29.278826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.278776 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-certs\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.279115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.279075 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d53f41f-fb95-473f-981f-a0752c78dd8d-istiod-ca-cert\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.279470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.279447 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-socket\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.279714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.279693 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqwk\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-kube-api-access-stqwk\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.279954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.279934 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-envoy\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.280195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.280173 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-data\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.316637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.316587 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-credential-socket\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.317064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.317036 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-token\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.317335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.317303 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-podinfo\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.317809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.317787 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzcs8\" (UniqueName: \"kubernetes.io/projected/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc-kube-api-access-jzcs8\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:29.356164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.349272 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01f7ce3e-nqpjv"] Nov 04 01:25:29.382225 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.359199 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-62510d3b-openshift-default-dockercfg-r6ct6\"" Nov 04 01:25:29.478354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.474280 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:29.526386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.517618 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-podinfo\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.526386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.517903 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-certs\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.526386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.517941 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d53f41f-fb95-473f-981f-a0752c78dd8d-istiod-ca-cert\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.526386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.518089 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-socket\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.533759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.533720 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqwk\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-kube-api-access-stqwk\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.534199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.534174 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-envoy\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.534533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.534503 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-data\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.608576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.608532 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:29.609294 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.609262 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:25:29.610167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.610142 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:29.627879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.627486 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:29.628505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.628459 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:29.647677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.647594 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:29.696170 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.671616 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-socket\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.720148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.713522 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-data\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.720148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.713627 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-certs\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.749156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.734220 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d53f41f-fb95-473f-981f-a0752c78dd8d-istiod-ca-cert\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.783081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.783043 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-credential-socket\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.783273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.783200 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-token\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.795891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.795743 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-credential-socket\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.804939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.804904 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:29.883155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.831221 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:29.883155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.848274 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-podinfo\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:29.883155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:29.861186 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:30.132146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.128699 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-envoy\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:30.147135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.144374 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-token\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:30.163169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.154250 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqwk\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-kube-api-access-stqwk\") pod \"e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:30.174163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.170402 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2862d1b-f394-4a99-8b7d-1cf92a506326" path="/var/lib/kubelet/pods/f2862d1b-f394-4a99-8b7d-1cf92a506326/volumes" Nov 04 01:25:30.195981 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.195853 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4"] Nov 04 01:25:30.300526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.300438 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:30.305142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.303655 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:25:30.305623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.305598 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.317676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.317655 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-296c124f-openshift-default-dockercfg-vg5v7\"" Nov 04 01:25:30.660187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.647212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:30.674521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.657212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:30.675251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.657263 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:25:30.675648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.658043 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" event={"ID":"82e10c2c-739e-441c-9a7a-9385e2f0d26f","Type":"ContainerStarted","Data":"6c51764c64c3aa35f6928fdee7b2e1562000a160b517f34528cd37fd9361072c"} Nov 04 01:25:30.675818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.673441 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:25:30.676381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.673478 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:25:30.709154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.704783 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:30.852515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.852471 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwh8d\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-kube-api-access-bwh8d\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.852834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.852806 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47cf65d-ee47-4303-baad-8911456e7299-istiod-ca-cert\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.861145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.853104 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-data\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.896488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.896453 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-certs\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.933354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.933258 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-envoy\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.980211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.980172 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47cf65d-ee47-4303-baad-8911456e7299-istio-podinfo\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.980699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.980659 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-credential-socket\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.980929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.980908 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-istio-token\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.981132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.981101 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-socket\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:30.981552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.891586 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:30.982078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:30.895014 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.139545 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-socket\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.139677 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwh8d\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-kube-api-access-bwh8d\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.139715 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47cf65d-ee47-4303-baad-8911456e7299-istiod-ca-cert\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.139807 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-data\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.139898 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-certs\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.139969 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-envoy\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.140034 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47cf65d-ee47-4303-baad-8911456e7299-istio-podinfo\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.140241 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-credential-socket\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.140272 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-istio-token\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.140993 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-socket\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.141866 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47cf65d-ee47-4303-baad-8911456e7299-istiod-ca-cert\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.142115 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-data\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.143155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.142362 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-certs\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.168797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.168764 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-envoy\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.227196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.218427 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-credential-socket\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.289912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.278622 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-istio-token\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.289912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.283314 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:31.309017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.308983 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwh8d\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-kube-api-access-bwh8d\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.610171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.602219 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:31.610405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.610359 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4"] Nov 04 01:25:31.610496 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.610481 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:25:31.610554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.610534 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:25:31.636279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.624625 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5"] Nov 04 01:25:31.639049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.639020 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8276ec5-9j2k5"] Nov 04 01:25:31.639272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.639257 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk"] Nov 04 01:25:31.685145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.640986 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47cf65d-ee47-4303-baad-8911456e7299-istio-podinfo\") pod \"e2e-ait-296c124f-openshift-default-687f659d99-xfwb4\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:31.685145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.678339 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:31.685145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.680206 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:25:31.685145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.680489 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:25:31.727168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.652302 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:25:31.727454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:31.727433 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e36c1537-hdphk"] Nov 04 01:25:32.041177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.033660 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:25:32.289143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.265410 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:32.298557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.298455 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:32.328302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.328267 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:25:32.402582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.402357 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:32.486140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.470374 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2"] Nov 04 01:25:32.486140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.470536 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:25:32.761295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.761258 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:25:32.838729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.838684 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:32.936646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.936614 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:25:33.014803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.014375 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:25:33.017700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:32.953917 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:33.031150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.021963 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:25:33.031150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.022734 5051 generic.go:356] "Generic (PLEG): container finished" podID="82e10c2c-739e-441c-9a7a-9385e2f0d26f" containerID="25f7e6064d4882603f005ab167c36cf7b67488de00831be1d98ab26d7173841c" exitCode=0 Nov 04 01:25:33.031150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.024653 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" event={"ID":"82e10c2c-739e-441c-9a7a-9385e2f0d26f","Type":"ContainerDied","Data":"25f7e6064d4882603f005ab167c36cf7b67488de00831be1d98ab26d7173841c"} Nov 04 01:25:33.031150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.025764 5051 patch_prober.go:29] interesting pod/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:25:33.031150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.025808 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" podUID="10b73b39-ac9b-4d1d-ae58-feb773882733" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:25:33.039554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.039530 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:25:33.469522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.467908 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:25:33.594106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.592649 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:33.594106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.592768 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:33.594106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.592849 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:33.604163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.603162 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c59d196-d8d8-451c-bd32-61657f80304f" path="/var/lib/kubelet/pods/5c59d196-d8d8-451c-bd32-61657f80304f/volumes" Nov 04 01:25:33.604163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.603894 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8359168a-65f4-4ba8-972b-608282de3f61" path="/var/lib/kubelet/pods/8359168a-65f4-4ba8-972b-608282de3f61/volumes" Nov 04 01:25:33.604493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.604467 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:25:33.670205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.670171 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk"] Nov 04 01:25:33.716442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.716405 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:25:33.724276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.723949 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7b10de73-6c59fcf75d-bmtxk"] Nov 04 01:25:33.853015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.849333 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4"] Nov 04 01:25:33.873855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.861480 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx"] Nov 04 01:25:33.891416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.874783 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-774d6cd7-xvzq4"] Nov 04 01:25:33.914715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:33.892912 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3cf45bd-vtfbx"] Nov 04 01:25:34.000241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.000136 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf"] Nov 04 01:25:34.047360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.047324 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" event={"ID":"0d53f41f-fb95-473f-981f-a0752c78dd8d","Type":"ContainerStarted","Data":"76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca"} Nov 04 01:25:34.047509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.047386 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" event={"ID":"0d53f41f-fb95-473f-981f-a0752c78dd8d","Type":"ContainerStarted","Data":"8526552ad01881e06ac2a982a8c8c5c29bced7c07245a9b56c74b284dc077739"} Nov 04 01:25:34.090200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.090096 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" podStartSLOduration=10.090077835 podStartE2EDuration="10.090077835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:34.086073624 +0000 UTC m=+1673.057822518" watchObservedRunningTime="2026-11-04 01:25:34.090077835 +0000 UTC m=+1673.061826707" Nov 04 01:25:34.140184 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:25:34.140108 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47cf65d_ee47_4303_baad_8911456e7299.slice/crio-df495e63a14ca5e85ccecaf1791bd2cd2ef9389e8dbbb6bd283a5d98cf73558d WatchSource:0}: Error finding container df495e63a14ca5e85ccecaf1791bd2cd2ef9389e8dbbb6bd283a5d98cf73558d: Status 404 returned error can't find the container with id df495e63a14ca5e85ccecaf1791bd2cd2ef9389e8dbbb6bd283a5d98cf73558d Nov 04 01:25:34.141891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.141862 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4"] Nov 04 01:25:34.144804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.144256 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:34.144804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.144338 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:34.144804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.144412 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:25:34.175902 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:34.175823 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:25:34.350153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.349974 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:34.396189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.374301 5051 patch_prober.go:29] interesting pod/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body= Nov 04 01:25:34.396189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.374367 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" podUID="0d53f41f-fb95-473f-981f-a0752c78dd8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" Nov 04 01:25:34.617144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.614469 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:34.706369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.703978 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca\") pod \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " Nov 04 01:25:34.706369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.704131 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v\" (UniqueName: \"kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v\") pod \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " Nov 04 01:25:34.706369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.704331 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca\") pod \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\" (UID: \"82e10c2c-739e-441c-9a7a-9385e2f0d26f\") " Nov 04 01:25:34.706369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.705273 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca" pod "82e10c2c-739e-441c-9a7a-9385e2f0d26f" (UID: "82e10c2c-739e-441c-9a7a-9385e2f0d26f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:34.720025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.719766 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v" pod "82e10c2c-739e-441c-9a7a-9385e2f0d26f" (UID: "82e10c2c-739e-441c-9a7a-9385e2f0d26f"). InnerVolumeSpecName "kube-api-access-ng45v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:34.723854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.723815 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca" pod "82e10c2c-739e-441c-9a7a-9385e2f0d26f" (UID: "82e10c2c-739e-441c-9a7a-9385e2f0d26f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:25:34.806728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.806639 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:34.806728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.806675 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng45v\" (UniqueName: \"kubernetes.io/projected/82e10c2c-739e-441c-9a7a-9385e2f0d26f-kube-api-access-ng45v\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:34.806728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:34.806695 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e10c2c-739e-441c-9a7a-9385e2f0d26f-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:35.073142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.064282 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" Nov 04 01:25:35.073142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.064286 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2" event={"ID":"82e10c2c-739e-441c-9a7a-9385e2f0d26f","Type":"ContainerDied","Data":"6c51764c64c3aa35f6928fdee7b2e1562000a160b517f34528cd37fd9361072c"} Nov 04 01:25:35.073142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.064751 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c51764c64c3aa35f6928fdee7b2e1562000a160b517f34528cd37fd9361072c" Nov 04 01:25:35.073142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.069258 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" event={"ID":"b47cf65d-ee47-4303-baad-8911456e7299","Type":"ContainerStarted","Data":"de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89"} Nov 04 01:25:35.073142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.069322 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" event={"ID":"b47cf65d-ee47-4303-baad-8911456e7299","Type":"ContainerStarted","Data":"df495e63a14ca5e85ccecaf1791bd2cd2ef9389e8dbbb6bd283a5d98cf73558d"} Nov 04 01:25:35.103187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.103091 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" podStartSLOduration=10.103072643 podStartE2EDuration="10.103072643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:35.091137755 +0000 UTC m=+1674.062886621" watchObservedRunningTime="2026-11-04 01:25:35.103072643 +0000 UTC m=+1674.074821511" Nov 04 01:25:35.450702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.450555 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ebe1b7-5a95-455c-91a7-a49694d43109" path="/var/lib/kubelet/pods/24ebe1b7-5a95-455c-91a7-a49694d43109/volumes" Nov 04 01:25:35.451474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.451427 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc" path="/var/lib/kubelet/pods/b56cb3c5-2ef2-4d62-8d57-c0bd60beabfc/volumes" Nov 04 01:25:35.452359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.452325 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e120ca74-3b6f-4241-a8a7-2ce9fd3d0571" path="/var/lib/kubelet/pods/e120ca74-3b6f-4241-a8a7-2ce9fd3d0571/volumes" Nov 04 01:25:35.452939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.452914 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:35.473816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.473784 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:35.480045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:35.480018 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:36.081622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:36.081578 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:36.082093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:36.081639 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:36.082771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:36.082647 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:25:36.083114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:36.083094 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:25:38.285162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:38.283905 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc"] Nov 04 01:25:38.318180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:38.315809 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cddeac7-8vdhc"] Nov 04 01:25:38.730336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:38.720557 5051 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:38.730336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:38.720638 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" podUID="c84934a5-2c51-47d8-a0d7-929640068888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:39.511859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:39.501674 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cf6ade-85bd-4c25-a923-4c041c046941" path="/var/lib/kubelet/pods/e9cf6ade-85bd-4c25-a923-4c041c046941/volumes" Nov 04 01:25:40.040455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:40.040421 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97"] Nov 04 01:25:40.048730 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:40.048693 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea2b688-sjp97"] Nov 04 01:25:41.319140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:41.319085 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac097ca4-142b-4dbc-8764-0ff9f2b5427b" path="/var/lib/kubelet/pods/ac097ca4-142b-4dbc-8764-0ff9f2b5427b/volumes" Nov 04 01:25:43.041317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:43.041234 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5"] Nov 04 01:25:43.051308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:43.051277 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cbc7d-v28b5"] Nov 04 01:25:43.192150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:43.192106 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab" path="/var/lib/kubelet/pods/a0a2fed7-5c0d-4b3b-98be-9928ccc7f0ab/volumes" Nov 04 01:25:45.142146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:45.071532 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl"] Nov 04 01:25:45.142146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:45.126564 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfe3388-fnvdl"] Nov 04 01:25:45.298147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:45.296421 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b39b24-5d37-4443-b5aa-6efd51aa3b04" path="/var/lib/kubelet/pods/b1b39b24-5d37-4443-b5aa-6efd51aa3b04/volumes" Nov 04 01:25:47.104802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:47.104730 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2"] Nov 04 01:25:47.115260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:47.115226 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7024f0c-4q9b2"] Nov 04 01:25:47.362514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:47.362396 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e10c2c-739e-441c-9a7a-9385e2f0d26f" path="/var/lib/kubelet/pods/82e10c2c-739e-441c-9a7a-9385e2f0d26f/volumes" Nov 04 01:25:47.465428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:47.462469 5051 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" secret="" err="secret \"e2e-ait-d7024f0c-openshift-default-dockercfg-bzddd\" not found" Nov 04 01:25:47.564981 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:47.563774 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg"] Nov 04 01:25:48.041110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:48.041035 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" podUID="c84934a5-2c51-47d8-a0d7-929640068888" containerName="istio-proxy" containerID="cri-o://9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192" gracePeriod=30 Nov 04 01:25:48.604808 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:48.604639 5051 patch_prober.go:29] interesting pod/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:48.604808 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:48.604730 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" podUID="10b73b39-ac9b-4d1d-ae58-feb773882733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:50.118152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:50.116541 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9"] Nov 04 01:25:50.144615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:50.144060 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee1c1373-9d8q9"] Nov 04 01:25:51.193650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:51.193608 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2f6b72-cb78-41d0-ac5a-e5d302a713b9" path="/var/lib/kubelet/pods/4a2f6b72-cb78-41d0-ac5a-e5d302a713b9/volumes" Nov 04 01:25:53.103038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:53.102997 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4"] Nov 04 01:25:53.116033 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:53.116002 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8daa67e6-grqt4"] Nov 04 01:25:53.314838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:53.314800 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f8ed22-7c81-47b6-a0fc-d52b366f305f" path="/var/lib/kubelet/pods/48f8ed22-7c81-47b6-a0fc-d52b366f305f/volumes" Nov 04 01:25:53.698258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:53.698230 5051 patch_prober.go:29] interesting pod/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:25:53.698411 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:53.698289 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" podUID="c84934a5-2c51-47d8-a0d7-929640068888" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:25:54.321345 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.321311 5051 generic.go:356] "Generic (PLEG): container finished" podID="10b73b39-ac9b-4d1d-ae58-feb773882733" containerID="5a02099e879a51bd63614a5f2dbd94491b669d05e408de5df7f8a9c331d25d6a" exitCode=0 Nov 04 01:25:54.321759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.321360 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" event={"ID":"10b73b39-ac9b-4d1d-ae58-feb773882733","Type":"ContainerDied","Data":"5a02099e879a51bd63614a5f2dbd94491b669d05e408de5df7f8a9c331d25d6a"} Nov 04 01:25:54.889807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.889776 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:25:54.959980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.959567 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-podinfo\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.960141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.960056 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-certs\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.960346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.960286 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-envoy\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.960445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.960347 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-credential-socket\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.960445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.960378 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-socket\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.960554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.960451 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-token\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-token\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.960554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.960501 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-kube-api-access-x9zlf\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-kube-api-access-x9zlf\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.960670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.960570 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b73b39-ac9b-4d1d-ae58-feb773882733-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b73b39-ac9b-4d1d-ae58-feb773882733-istiod-ca-cert\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.960670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.960658 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-data\") pod \"10b73b39-ac9b-4d1d-ae58-feb773882733\" (UID: \"10b73b39-ac9b-4d1d-ae58-feb773882733\") " Nov 04 01:25:54.962072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.961471 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-credential-socket" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:54.962072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.961785 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-data" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:54.962072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.961804 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b73b39-ac9b-4d1d-ae58-feb773882733-istiod-ca-cert" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:54.962072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.961996 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-certs" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:54.962072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.962027 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:54.962454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.962175 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-socket" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:54.968784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.968752 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-token" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:54.968897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.968788 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-kube-api-access-x9zlf" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "kube-api-access-x9zlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:54.968897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.968863 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-podinfo" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:25:54.969242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:54.969222 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-envoy" pod "10b73b39-ac9b-4d1d-ae58-feb773882733" (UID: "10b73b39-ac9b-4d1d-ae58-feb773882733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:55.064422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.064330 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.064422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.064370 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.064422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.064384 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9zlf\" (UniqueName: \"kubernetes.io/projected/10b73b39-ac9b-4d1d-ae58-feb773882733-kube-api-access-x9zlf\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.064422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.064396 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b73b39-ac9b-4d1d-ae58-feb773882733-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.064422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.064409 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.064422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.064423 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.065541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.064437 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.065541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.064448 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b73b39-ac9b-4d1d-ae58-feb773882733-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.192634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.192601 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:25:55.267970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.267934 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84934a5-2c51-47d8-a0d7-929640068888-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84934a5-2c51-47d8-a0d7-929640068888-istiod-ca-cert\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.268173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.268018 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-socket\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.268173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.268081 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-istio-token\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-istio-token\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.268173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.268111 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-credential-socket\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.268372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.268175 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-data\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.268372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.268282 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-kube-api-access-h86md\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-kube-api-access-h86md\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.268483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.268383 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c84934a5-2c51-47d8-a0d7-929640068888-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84934a5-2c51-47d8-a0d7-929640068888-istio-podinfo\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.268628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.268609 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-certs\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.268702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.268652 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-envoy\") pod \"c84934a5-2c51-47d8-a0d7-929640068888\" (UID: \"c84934a5-2c51-47d8-a0d7-929640068888\") " Nov 04 01:25:55.270276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.269972 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-socket" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:55.270276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.269985 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-certs" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:55.270276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.270080 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84934a5-2c51-47d8-a0d7-929640068888-istiod-ca-cert" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:25:55.270276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.270107 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-credential-socket" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:55.270276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.270241 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-data" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:55.272341 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.272307 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-envoy" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:25:55.272699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.272665 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-istio-token" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:55.273188 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.273154 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c84934a5-2c51-47d8-a0d7-929640068888-istio-podinfo" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:25:55.273968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.273942 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-kube-api-access-h86md" pod "c84934a5-2c51-47d8-a0d7-929640068888" (UID: "c84934a5-2c51-47d8-a0d7-929640068888"). InnerVolumeSpecName "kube-api-access-h86md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:25:55.332362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.332323 5051 generic.go:356] "Generic (PLEG): container finished" podID="c84934a5-2c51-47d8-a0d7-929640068888" containerID="9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192" exitCode=0 Nov 04 01:25:55.332789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.332384 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" Nov 04 01:25:55.332789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.332415 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" event={"ID":"c84934a5-2c51-47d8-a0d7-929640068888","Type":"ContainerDied","Data":"9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192"} Nov 04 01:25:55.332789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.332488 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg" event={"ID":"c84934a5-2c51-47d8-a0d7-929640068888","Type":"ContainerDied","Data":"030d00bcc99757b7c2d2afe8608f6dfc42018d3cdc05108970871b0585ff4643"} Nov 04 01:25:55.332789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.332526 5051 scope.go:122] "RemoveContainer" containerID="9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192" Nov 04 01:25:55.334331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.334301 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" Nov 04 01:25:55.334423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.334294 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc" event={"ID":"10b73b39-ac9b-4d1d-ae58-feb773882733","Type":"ContainerDied","Data":"19592f583a10359d3eae2834a5fc860140a5b6dc9d0d1023f692f4f504ff8d39"} Nov 04 01:25:55.346819 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.346800 5051 scope.go:122] "RemoveContainer" containerID="9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192" Nov 04 01:25:55.347103 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:25:55.347085 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192\": container with ID starting with 9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192 not found: ID does not exist" containerID="9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192" Nov 04 01:25:55.347247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.347115 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192"} err="failed to get container status \"9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192\": rpc error: code = NotFound desc = could not find container \"9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192\": container with ID starting with 9c2d18f76ca6490616eac4cf661fd3b90194ccce9dc5c841a1244bdb4e0ec192 not found: ID does not exist" Nov 04 01:25:55.347247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.347154 5051 scope.go:122] "RemoveContainer" containerID="5a02099e879a51bd63614a5f2dbd94491b669d05e408de5df7f8a9c331d25d6a" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404664 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c84934a5-2c51-47d8-a0d7-929640068888-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404702 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404719 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404733 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c84934a5-2c51-47d8-a0d7-929640068888-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404748 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404762 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404775 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404790 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c84934a5-2c51-47d8-a0d7-929640068888-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.406152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.404804 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86md\" (UniqueName: \"kubernetes.io/projected/c84934a5-2c51-47d8-a0d7-929640068888-kube-api-access-h86md\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:25:55.481169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.476196 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc"] Nov 04 01:25:55.543164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.542186 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7b10de73-gw-openshift-default-bfbb8bd77-gbldc"] Nov 04 01:25:55.664986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.664955 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg"] Nov 04 01:25:55.717200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:55.715237 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d7024f0c-openshift-default-8c87b6dbf-tqxwg"] Nov 04 01:25:56.089025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:56.088983 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj"] Nov 04 01:25:56.100429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:56.100396 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cdb602-hn9bj"] Nov 04 01:25:57.191287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.191251 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b73b39-ac9b-4d1d-ae58-feb773882733" path="/var/lib/kubelet/pods/10b73b39-ac9b-4d1d-ae58-feb773882733/volumes" Nov 04 01:25:57.191949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.191926 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d8a0be-a45e-421e-816e-9e11997d83e1" path="/var/lib/kubelet/pods/52d8a0be-a45e-421e-816e-9e11997d83e1/volumes" Nov 04 01:25:57.192586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.192562 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84934a5-2c51-47d8-a0d7-929640068888" path="/var/lib/kubelet/pods/c84934a5-2c51-47d8-a0d7-929640068888/volumes" Nov 04 01:25:57.752910 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.752876 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq"] Nov 04 01:25:57.753642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753622 5051 cpu_manager.go:414] "containerMap: removing container" podUID="10b73b39-ac9b-4d1d-ae58-feb773882733" containerName="istio-proxy" Nov 04 01:25:57.753642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753639 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b73b39-ac9b-4d1d-ae58-feb773882733" containerName="istio-proxy" Nov 04 01:25:57.753823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753665 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c84934a5-2c51-47d8-a0d7-929640068888" containerName="istio-proxy" Nov 04 01:25:57.753823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753671 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84934a5-2c51-47d8-a0d7-929640068888" containerName="istio-proxy" Nov 04 01:25:57.753823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753682 5051 cpu_manager.go:414] "containerMap: removing container" podUID="82e10c2c-739e-441c-9a7a-9385e2f0d26f" containerName="revoke-keys" Nov 04 01:25:57.753823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753687 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e10c2c-739e-441c-9a7a-9385e2f0d26f" containerName="revoke-keys" Nov 04 01:25:57.753823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753792 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84934a5-2c51-47d8-a0d7-929640068888" containerName="istio-proxy" Nov 04 01:25:57.753823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753800 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e10c2c-739e-441c-9a7a-9385e2f0d26f" containerName="revoke-keys" Nov 04 01:25:57.753823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.753818 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b73b39-ac9b-4d1d-ae58-feb773882733" containerName="istio-proxy" Nov 04 01:25:57.759115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.759091 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.767565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.765074 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:25:57.767565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.765174 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:25:57.767565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.765229 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:25:57.767565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.766114 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:25:57.769771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.769747 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq"] Nov 04 01:25:57.807421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.807264 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.807592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.807568 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjzq\" (UniqueName: \"kubernetes.io/projected/aa9001b3-4896-4b4e-977d-86d5cc183d6c-kube-api-access-crjzq\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.807669 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.807636 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.909742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.909710 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.909906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.909836 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjzq\" (UniqueName: \"kubernetes.io/projected/aa9001b3-4896-4b4e-977d-86d5cc183d6c-kube-api-access-crjzq\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.909906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.909871 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.910621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.910599 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.912726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.912706 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:57.920387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:57.920357 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjzq\" (UniqueName: \"kubernetes.io/projected/aa9001b3-4896-4b4e-977d-86d5cc183d6c-kube-api-access-crjzq\") pod \"maas-api-revoke-keys-e2e-ait-296c124f-kwjsq\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:58.092888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:58.092802 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:25:58.517100 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:25:58.517067 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9001b3_4896_4b4e_977d_86d5cc183d6c.slice/crio-9f1a35f7433345841f53be55007239ebdacd23efd7509d25f9afbf9a879311a4 WatchSource:0}: Error finding container 9f1a35f7433345841f53be55007239ebdacd23efd7509d25f9afbf9a879311a4: Status 404 returned error can't find the container with id 9f1a35f7433345841f53be55007239ebdacd23efd7509d25f9afbf9a879311a4 Nov 04 01:25:58.517572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:58.517538 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq"] Nov 04 01:25:59.451831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:59.451718 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" event={"ID":"aa9001b3-4896-4b4e-977d-86d5cc183d6c","Type":"ContainerStarted","Data":"51c3d01fee84bae3fe0148dcddc7ed700d3f9988e95fccf46cbe5c3d52080498"} Nov 04 01:25:59.451831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:59.451796 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" event={"ID":"aa9001b3-4896-4b4e-977d-86d5cc183d6c","Type":"ContainerStarted","Data":"9f1a35f7433345841f53be55007239ebdacd23efd7509d25f9afbf9a879311a4"} Nov 04 01:25:59.477447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:25:59.477388 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" podStartSLOduration=2.477363861 podStartE2EDuration="2.477363861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:25:59.469188011 +0000 UTC m=+1698.440936883" watchObservedRunningTime="2026-11-04 01:25:59.477363861 +0000 UTC m=+1698.449112732" Nov 04 01:26:00.464948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:00.464908 5051 generic.go:356] "Generic (PLEG): container finished" podID="aa9001b3-4896-4b4e-977d-86d5cc183d6c" containerID="51c3d01fee84bae3fe0148dcddc7ed700d3f9988e95fccf46cbe5c3d52080498" exitCode=0 Nov 04 01:26:00.465406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:00.464989 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" event={"ID":"aa9001b3-4896-4b4e-977d-86d5cc183d6c","Type":"ContainerDied","Data":"51c3d01fee84bae3fe0148dcddc7ed700d3f9988e95fccf46cbe5c3d52080498"} Nov 04 01:26:02.220579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.220553 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:26:02.277697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.275855 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-xks-ca\") pod \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " Nov 04 01:26:02.277697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.276179 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9001b3-4896-4b4e-977d-86d5cc183d6c-kube-api-access-crjzq\" (UniqueName: \"kubernetes.io/projected/aa9001b3-4896-4b4e-977d-86d5cc183d6c-kube-api-access-crjzq\") pod \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " Nov 04 01:26:02.277697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.276260 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-openshift-ca\") pod \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\" (UID: \"aa9001b3-4896-4b4e-977d-86d5cc183d6c\") " Nov 04 01:26:02.279795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.279407 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-openshift-ca" pod "aa9001b3-4896-4b4e-977d-86d5cc183d6c" (UID: "aa9001b3-4896-4b4e-977d-86d5cc183d6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:02.309179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.306365 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-xks-ca" pod "aa9001b3-4896-4b4e-977d-86d5cc183d6c" (UID: "aa9001b3-4896-4b4e-977d-86d5cc183d6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:02.316705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.316670 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9001b3-4896-4b4e-977d-86d5cc183d6c-kube-api-access-crjzq" pod "aa9001b3-4896-4b4e-977d-86d5cc183d6c" (UID: "aa9001b3-4896-4b4e-977d-86d5cc183d6c"). InnerVolumeSpecName "kube-api-access-crjzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:02.380409 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.380377 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjzq\" (UniqueName: \"kubernetes.io/projected/aa9001b3-4896-4b4e-977d-86d5cc183d6c-kube-api-access-crjzq\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:02.380409 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.380406 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:02.380409 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.380417 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9001b3-4896-4b4e-977d-86d5cc183d6c-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:02.537028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.536997 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" Nov 04 01:26:02.537251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.537034 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq" event={"ID":"aa9001b3-4896-4b4e-977d-86d5cc183d6c","Type":"ContainerDied","Data":"9f1a35f7433345841f53be55007239ebdacd23efd7509d25f9afbf9a879311a4"} Nov 04 01:26:02.537251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:02.537089 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f1a35f7433345841f53be55007239ebdacd23efd7509d25f9afbf9a879311a4" Nov 04 01:26:04.154530 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:26:04.154484 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:26:05.903147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:05.899231 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2"] Nov 04 01:26:05.903147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:05.900380 5051 cpu_manager.go:414] "containerMap: removing container" podUID="aa9001b3-4896-4b4e-977d-86d5cc183d6c" containerName="revoke-keys" Nov 04 01:26:05.903147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:05.900404 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9001b3-4896-4b4e-977d-86d5cc183d6c" containerName="revoke-keys" Nov 04 01:26:05.903147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:05.900657 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9001b3-4896-4b4e-977d-86d5cc183d6c" containerName="revoke-keys" Nov 04 01:26:06.235394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.235358 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2"] Nov 04 01:26:06.236937 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.236911 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:06.311143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.281580 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:26:06.311143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.282722 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:26:06.311143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.282783 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:26:06.311143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.282868 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:26:06.404271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.404227 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:06.404641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.404591 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:06.404833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.404814 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckbvg\" (UniqueName: \"kubernetes.io/projected/005ed678-4d95-42e3-baf4-138d972d095a-kube-api-access-ckbvg\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:06.505320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.475175 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld"] Nov 04 01:26:06.568040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.567967 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:06.568335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.568064 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:06.568335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.568149 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckbvg\" (UniqueName: \"kubernetes.io/projected/005ed678-4d95-42e3-baf4-138d972d095a-kube-api-access-ckbvg\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:06.583220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.578319 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld"] Nov 04 01:26:06.583220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.578451 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:06.669783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.669744 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:06.694147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.671273 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:06.713805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.698295 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:06.731159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.714843 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:06.732040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.732007 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:06.732513 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.732490 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:06.732746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.732726 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9a5635-0458-44df-8272-31bb8a86c81c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:06.732931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.732915 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:06.733162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.733145 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kttw\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-kube-api-access-4kttw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:06.733418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.733401 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:06.877199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.835588 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.027582 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9a5635-0458-44df-8272-31bb8a86c81c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.027634 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.027705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kttw\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-kube-api-access-4kttw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.027808 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.028039 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.028197 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:06.852094 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.029241 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.042401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.034581 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckbvg\" (UniqueName: \"kubernetes.io/projected/005ed678-4d95-42e3-baf4-138d972d095a-kube-api-access-ckbvg\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:07.044561 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.044520 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.106224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.087717 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.170212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.138974 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.232951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.219280 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:07.232951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.222785 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kttw\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-kube-api-access-4kttw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.253146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.234612 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.253146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.234892 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:07.253146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.241851 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9a5635-0458-44df-8272-31bb8a86c81c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fjkld\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.272337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.263415 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:07.347165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.345111 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:07.697296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.697245 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:07.746679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.718891 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:26:07.746679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.720540 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:26:07.853019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:07.852970 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:08.324680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.024388 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:08.572222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.570895 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:08.618254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.598793 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:08.657055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.657010 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:08.657782 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.657755 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:26:08.726766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.726729 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:26:08.727212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.727193 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:26:08.735684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.735566 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:08.792746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.792701 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:08.855792 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:08.855663 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:09.333354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:09.333301 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:09.584594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:09.584549 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:26:09.585027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:09.584999 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 04 01:26:09.585366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:09.585343 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:26:09.612386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:09.612342 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:09.669147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:09.657971 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:10.014186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:09.993314 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:26:10.206046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:10.206000 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:26:10.494201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:10.305831 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:10.494201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:10.345222 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:26:10.494201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:10.413240 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:26:10.640629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:10.594667 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:10.641835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:10.641782 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:10.789196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:10.731251 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:11.058081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.057989 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:11.155152 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:26:11.060010 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.878s" Nov 04 01:26:11.278827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.230568 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:26:11.278827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.232848 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:26:11.278827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.233716 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:26:11.383925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.383823 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:26:11.486106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.486071 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:11.539603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.539572 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:11.544215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.509622 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:26:11.571287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.508150 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Nov 04 01:26:12.071141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.509736 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:26:12.071141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.510101 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.539482 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:26:11.490382 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005ed678_4d95_42e3_baf4_138d972d095a.slice/crio-854e962379fedf80b805a981bcb7d0405263d815519b5dd7b75585cb92b1724b WatchSource:0}: Error finding container 854e962379fedf80b805a981bcb7d0405263d815519b5dd7b75585cb92b1724b: Status 404 returned error can't find the container with id 854e962379fedf80b805a981bcb7d0405263d815519b5dd7b75585cb92b1724b Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.506236 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:12.108871 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.659697 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.582788 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:12.125532 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.509650 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:11.812203 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:26:12.128955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:12.127981 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:12.646208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:12.643235 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:26:12.826661 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:12.826610 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" podUID="f80b3f88-24db-4e50-aecc-c1c19664e0ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 04 01:26:12.912591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:12.905989 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:26:13.255267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:13.249786 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:13.255267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:13.250490 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:13.255267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:13.250936 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:13.255267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:13.251301 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:26:13.270409 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:13.270342 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:13.348163 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:26:13.339364 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.106s" Nov 04 01:26:13.539028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:13.535389 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:26:13.778697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:13.778649 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:14.105955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.105905 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:14.297580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.297545 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:26:14.297898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.297871 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" event={"ID":"bc9a5635-0458-44df-8272-31bb8a86c81c","Type":"ContainerStarted","Data":"c17768d4ed3128d5cea4085639f33deb927d76b634e08b28caa738e84c3d8411"} Nov 04 01:26:14.298178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.298159 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:26:14.298398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.298383 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:26:14.298567 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.298554 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:26:14.298693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.298678 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5"] Nov 04 01:26:14.298803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.298793 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab1ea1a8-p6rq5"] Nov 04 01:26:14.298916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.298906 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld"] Nov 04 01:26:14.502232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.502190 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2"] Nov 04 01:26:14.612145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.590642 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:14.660200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.600455 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq"] Nov 04 01:26:14.903147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.837416 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-296c124f-kwjsq"] Nov 04 01:26:14.903147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.864135 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:26:14.903147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.864354 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:14.903147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.865083 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:26:14.929480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.929386 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:26:14.999145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.997035 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:14.999145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.997364 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:26:14.999145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:14.997551 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:26:15.609164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.599187 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9001b3-4896-4b4e-977d-86d5cc183d6c" path="/var/lib/kubelet/pods/aa9001b3-4896-4b4e-977d-86d5cc183d6c/volumes" Nov 04 01:26:15.609164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.600019 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54fc5de-a9de-4a8a-88a8-473dff5f9cfc" path="/var/lib/kubelet/pods/f54fc5de-a9de-4a8a-88a8-473dff5f9cfc/volumes" Nov 04 01:26:15.609164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.600532 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4"] Nov 04 01:26:15.609164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.600755 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" podUID="b47cf65d-ee47-4303-baad-8911456e7299" containerName="istio-proxy" containerID="cri-o://de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89" gracePeriod=30 Nov 04 01:26:15.609164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.604419 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" event={"ID":"005ed678-4d95-42e3-baf4-138d972d095a","Type":"ContainerStarted","Data":"854e962379fedf80b805a981bcb7d0405263d815519b5dd7b75585cb92b1724b"} Nov 04 01:26:15.709657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.709611 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:26:15.824153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.822852 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:15.949648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.926229 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:26:15.949648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.926508 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:26:15.949648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.926727 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:26:15.949648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:15.926781 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:26:16.163377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.031335 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:26:16.236452 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.053601 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:16.324149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.053663 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:26:16.324149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.053681 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:26:16.539200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.524160 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" podStartSLOduration=11.524114131 podStartE2EDuration="11.524114131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:15.87699304 +0000 UTC m=+1714.848741912" watchObservedRunningTime="2026-11-04 01:26:16.524114131 +0000 UTC m=+1715.495863003" Nov 04 01:26:16.588209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.532880 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:26:16.711042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.710996 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:16.943205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.938403 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:17.002595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:16.964484 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:26:17.121701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.121649 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" event={"ID":"bc9a5635-0458-44df-8272-31bb8a86c81c","Type":"ContainerStarted","Data":"8ff0fc184255dadb352573957259b0d77e4191f2d04ff76e62accad08f2a9ffb"} Nov 04 01:26:17.140548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.140508 5051 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fjkld container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:26:17.140772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.140750 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" podUID="bc9a5635-0458-44df-8272-31bb8a86c81c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:26:17.142302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.142278 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:17.157149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.153657 5051 generic.go:356] "Generic (PLEG): container finished" podID="005ed678-4d95-42e3-baf4-138d972d095a" containerID="8e00a8fecfdbfd60d8946471d9e00511c86165468087b52e6c56687b8a0ade47" exitCode=0 Nov 04 01:26:17.157506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.157479 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" event={"ID":"005ed678-4d95-42e3-baf4-138d972d095a","Type":"ContainerDied","Data":"8e00a8fecfdbfd60d8946471d9e00511c86165468087b52e6c56687b8a0ade47"} Nov 04 01:26:17.227980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.227892 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:17.256971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.234786 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:26:17.330555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.330509 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:17.331211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.330803 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:17.484193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.481804 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:26:17.573255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:17.573226 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:26:18.016577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.015080 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:18.255201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.254563 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:18.266849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.266212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:26:18.267697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.267670 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:26:18.275160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.270428 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:26:18.340413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.340366 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:18.385551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.381473 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:26:18.480438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.472740 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:26:18.486161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.485299 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:26:18.628656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.607265 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:18.649218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.649192 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:26:18.760399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:18.760352 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:26:19.048288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.048034 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:26:19.120295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.112671 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:26:19.534724 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.534697 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:19.813490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.813461 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" Nov 04 01:26:19.813877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.813638 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2" event={"ID":"005ed678-4d95-42e3-baf4-138d972d095a","Type":"ContainerDied","Data":"854e962379fedf80b805a981bcb7d0405263d815519b5dd7b75585cb92b1724b"} Nov 04 01:26:19.814044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.814028 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="854e962379fedf80b805a981bcb7d0405263d815519b5dd7b75585cb92b1724b" Nov 04 01:26:19.855166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.854135 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005ed678-4d95-42e3-baf4-138d972d095a-kube-api-access-ckbvg\" (UniqueName: \"kubernetes.io/projected/005ed678-4d95-42e3-baf4-138d972d095a-kube-api-access-ckbvg\") pod \"005ed678-4d95-42e3-baf4-138d972d095a\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " Nov 04 01:26:19.855166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.854207 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-xks-ca\") pod \"005ed678-4d95-42e3-baf4-138d972d095a\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " Nov 04 01:26:19.855166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.854520 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-openshift-ca\") pod \"005ed678-4d95-42e3-baf4-138d972d095a\" (UID: \"005ed678-4d95-42e3-baf4-138d972d095a\") " Nov 04 01:26:19.870617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.870577 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-openshift-ca" pod "005ed678-4d95-42e3-baf4-138d972d095a" (UID: "005ed678-4d95-42e3-baf4-138d972d095a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:19.875842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.875810 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005ed678-4d95-42e3-baf4-138d972d095a-kube-api-access-ckbvg" pod "005ed678-4d95-42e3-baf4-138d972d095a" (UID: "005ed678-4d95-42e3-baf4-138d972d095a"). InnerVolumeSpecName "kube-api-access-ckbvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:19.885376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.885346 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-xks-ca" pod "005ed678-4d95-42e3-baf4-138d972d095a" (UID: "005ed678-4d95-42e3-baf4-138d972d095a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:19.957631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.957585 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckbvg\" (UniqueName: \"kubernetes.io/projected/005ed678-4d95-42e3-baf4-138d972d095a-kube-api-access-ckbvg\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.957631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.957624 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:19.957859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:19.957638 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/005ed678-4d95-42e3-baf4-138d972d095a-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:20.076976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:20.076888 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g"] Nov 04 01:26:20.097633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:20.087985 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f477b14-vb52g"] Nov 04 01:26:21.218271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:21.218236 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83864e06-2d7d-40c0-8d8c-48d8c58d3cdb" path="/var/lib/kubelet/pods/83864e06-2d7d-40c0-8d8c-48d8c58d3cdb/volumes" Nov 04 01:26:22.121343 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:22.121294 5051 patch_prober.go:29] interesting pod/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:22.121530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:22.121391 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" podUID="b47cf65d-ee47-4303-baad-8911456e7299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:22.122253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:22.122225 5051 patch_prober.go:29] interesting pod/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:22.122418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:22.122283 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" podUID="0d53f41f-fb95-473f-981f-a0752c78dd8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:29.168259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:29.168215 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2"] Nov 04 01:26:29.180762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:29.180046 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-62510d3b-b5wr2"] Nov 04 01:26:29.200037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:29.199985 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005ed678-4d95-42e3-baf4-138d972d095a" path="/var/lib/kubelet/pods/005ed678-4d95-42e3-baf4-138d972d095a/volumes" Nov 04 01:26:29.422185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:29.422084 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf"] Nov 04 01:26:29.422795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:29.422741 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" podUID="0d53f41f-fb95-473f-981f-a0752c78dd8d" containerName="istio-proxy" containerID="cri-o://76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca" gracePeriod=30 Nov 04 01:26:34.136744 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:26:34.136694 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:26:35.192074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.192034 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp"] Nov 04 01:26:35.192856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.192837 5051 cpu_manager.go:414] "containerMap: removing container" podUID="005ed678-4d95-42e3-baf4-138d972d095a" containerName="revoke-keys" Nov 04 01:26:35.192908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.192859 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005ed678-4d95-42e3-baf4-138d972d095a" containerName="revoke-keys" Nov 04 01:26:35.193028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.193016 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="005ed678-4d95-42e3-baf4-138d972d095a" containerName="revoke-keys" Nov 04 01:26:35.245459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.245421 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp"] Nov 04 01:26:35.245670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.245555 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.253188 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.250725 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:26:35.267897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.267870 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:26:35.268366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.268343 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:26:35.268490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.268462 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:26:35.366311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.366277 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.366509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.366359 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.366509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.366495 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbng\" (UniqueName: \"kubernetes.io/projected/c72b5492-f761-46d0-aa80-2c9abb6b0b81-kube-api-access-clbng\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.467642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.467552 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbng\" (UniqueName: \"kubernetes.io/projected/c72b5492-f761-46d0-aa80-2c9abb6b0b81-kube-api-access-clbng\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.467858 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.467841 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.467927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.467891 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.468820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.468779 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.470550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.470523 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.476580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.476560 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbng\" (UniqueName: \"kubernetes.io/projected/c72b5492-f761-46d0-aa80-2c9abb6b0b81-kube-api-access-clbng\") pod \"maas-api-revoke-sub-e067977e-2lpqp\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:35.656280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:35.656246 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:36.668887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:36.668842 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp"] Nov 04 01:26:37.032964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.032924 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m"] Nov 04 01:26:37.038346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.038312 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86667efb-cxk5m"] Nov 04 01:26:37.125351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.125268 5051 patch_prober.go:29] interesting pod/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:37.125503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.125359 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" podUID="0d53f41f-fb95-473f-981f-a0752c78dd8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:37.131321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.131283 5051 patch_prober.go:29] interesting pod/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:26:37.131440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.131337 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" podUID="b47cf65d-ee47-4303-baad-8911456e7299" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:26:37.191372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.191332 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c1988c6-4378-46cf-be31-172ee5be96d5" path="/var/lib/kubelet/pods/4c1988c6-4378-46cf-be31-172ee5be96d5/volumes" Nov 04 01:26:37.753703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.753330 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" event={"ID":"c72b5492-f761-46d0-aa80-2c9abb6b0b81","Type":"ContainerStarted","Data":"602338c09187215e2cc3312a2fa8eb4010f0cae79d35b82911b02fe49af653f2"} Nov 04 01:26:37.753703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.753398 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" event={"ID":"c72b5492-f761-46d0-aa80-2c9abb6b0b81","Type":"ContainerStarted","Data":"f13e4277de3477a6c6f9a873705691bb64c56c10a457d23f68788a7321e9056b"} Nov 04 01:26:37.775180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:37.775031 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" podStartSLOduration=2.77501187 podStartE2EDuration="2.77501187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:26:37.76946278 +0000 UTC m=+1736.741211652" watchObservedRunningTime="2026-11-04 01:26:37.77501187 +0000 UTC m=+1736.746760742" Nov 04 01:26:38.869150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:38.868597 5051 generic.go:356] "Generic (PLEG): container finished" podID="c72b5492-f761-46d0-aa80-2c9abb6b0b81" containerID="602338c09187215e2cc3312a2fa8eb4010f0cae79d35b82911b02fe49af653f2" exitCode=0 Nov 04 01:26:38.869150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:38.868699 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" event={"ID":"c72b5492-f761-46d0-aa80-2c9abb6b0b81","Type":"ContainerDied","Data":"602338c09187215e2cc3312a2fa8eb4010f0cae79d35b82911b02fe49af653f2"} Nov 04 01:26:39.146971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.146943 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:26:39.276175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276104 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-envoy\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.276366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276181 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-data\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.276366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276226 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-credential-socket\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.276366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276265 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-podinfo\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.276543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276526 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-socket\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.277309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276611 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-certs\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.277309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276673 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-kube-api-access-stqwk\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-kube-api-access-stqwk\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.277309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276761 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d53f41f-fb95-473f-981f-a0752c78dd8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d53f41f-fb95-473f-981f-a0752c78dd8d-istiod-ca-cert\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.277309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276809 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-token\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-token\") pod \"0d53f41f-fb95-473f-981f-a0752c78dd8d\" (UID: \"0d53f41f-fb95-473f-981f-a0752c78dd8d\") " Nov 04 01:26:39.277309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.276898 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-credential-socket" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.277309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.277093 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-data" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.277309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.277292 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-certs" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.277818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.277736 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.277818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.277757 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.277818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.277771 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.278352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.277820 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-socket" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.278737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.278714 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d53f41f-fb95-473f-981f-a0752c78dd8d-istiod-ca-cert" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:39.280437 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.280408 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-podinfo" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:39.281017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.280990 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-envoy" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:39.281342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.281317 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-kube-api-access-stqwk" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "kube-api-access-stqwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:39.281601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.281570 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-token" pod "0d53f41f-fb95-473f-981f-a0752c78dd8d" (UID: "0d53f41f-fb95-473f-981f-a0752c78dd8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:39.380233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.380191 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.380233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.380227 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqwk\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-kube-api-access-stqwk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.380486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.380244 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d53f41f-fb95-473f-981f-a0752c78dd8d-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.380486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.380258 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.380486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.380272 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.380486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.380287 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d53f41f-fb95-473f-981f-a0752c78dd8d-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:39.882973 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.882934 5051 generic.go:356] "Generic (PLEG): container finished" podID="0d53f41f-fb95-473f-981f-a0752c78dd8d" containerID="76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca" exitCode=0 Nov 04 01:26:39.883572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.882995 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" Nov 04 01:26:39.883572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.883020 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" event={"ID":"0d53f41f-fb95-473f-981f-a0752c78dd8d","Type":"ContainerDied","Data":"76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca"} Nov 04 01:26:39.883572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.883076 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf" event={"ID":"0d53f41f-fb95-473f-981f-a0752c78dd8d","Type":"ContainerDied","Data":"8526552ad01881e06ac2a982a8c8c5c29bced7c07245a9b56c74b284dc077739"} Nov 04 01:26:39.883572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.883103 5051 scope.go:122] "RemoveContainer" containerID="76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca" Nov 04 01:26:39.910298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.910195 5051 scope.go:122] "RemoveContainer" containerID="76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca" Nov 04 01:26:39.911529 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:26:39.911501 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca\": container with ID starting with 76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca not found: ID does not exist" containerID="76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca" Nov 04 01:26:39.911798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.911543 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca"} err="failed to get container status \"76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca\": rpc error: code = NotFound desc = could not find container \"76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca\": container with ID starting with 76289099ead0688d549dff4588b715ba8e9c84fd90e01acb7e0788d0b1b3e5ca not found: ID does not exist" Nov 04 01:26:39.918533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.918507 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf"] Nov 04 01:26:39.943828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:39.943796 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-62510d3b-openshift-default-7bd54dc97c-shpwf"] Nov 04 01:26:40.245180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.244849 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:40.304715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.304673 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-openshift-ca\") pod \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " Nov 04 01:26:40.304899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.304782 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72b5492-f761-46d0-aa80-2c9abb6b0b81-kube-api-access-clbng\" (UniqueName: \"kubernetes.io/projected/c72b5492-f761-46d0-aa80-2c9abb6b0b81-kube-api-access-clbng\") pod \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " Nov 04 01:26:40.305059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.305038 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-xks-ca\") pod \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\" (UID: \"c72b5492-f761-46d0-aa80-2c9abb6b0b81\") " Nov 04 01:26:40.305144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.305082 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-openshift-ca" pod "c72b5492-f761-46d0-aa80-2c9abb6b0b81" (UID: "c72b5492-f761-46d0-aa80-2c9abb6b0b81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:40.306105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.306038 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:40.308000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.307951 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-xks-ca" pod "c72b5492-f761-46d0-aa80-2c9abb6b0b81" (UID: "c72b5492-f761-46d0-aa80-2c9abb6b0b81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:26:40.308000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.307992 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72b5492-f761-46d0-aa80-2c9abb6b0b81-kube-api-access-clbng" pod "c72b5492-f761-46d0-aa80-2c9abb6b0b81" (UID: "c72b5492-f761-46d0-aa80-2c9abb6b0b81"). InnerVolumeSpecName "kube-api-access-clbng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:40.408087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.408007 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clbng\" (UniqueName: \"kubernetes.io/projected/c72b5492-f761-46d0-aa80-2c9abb6b0b81-kube-api-access-clbng\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:40.408087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.408049 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72b5492-f761-46d0-aa80-2c9abb6b0b81-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:40.919456 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.919416 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" event={"ID":"c72b5492-f761-46d0-aa80-2c9abb6b0b81","Type":"ContainerDied","Data":"f13e4277de3477a6c6f9a873705691bb64c56c10a457d23f68788a7321e9056b"} Nov 04 01:26:40.920136 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.920099 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13e4277de3477a6c6f9a873705691bb64c56c10a457d23f68788a7321e9056b" Nov 04 01:26:40.920328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:40.920314 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp" Nov 04 01:26:41.496160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:41.495529 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d53f41f-fb95-473f-981f-a0752c78dd8d" path="/var/lib/kubelet/pods/0d53f41f-fb95-473f-981f-a0752c78dd8d/volumes" Nov 04 01:26:42.418193 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:26:42.418115 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47cf65d_ee47_4303_baad_8911456e7299.slice/crio-de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47cf65d_ee47_4303_baad_8911456e7299.slice/crio-conmon-de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@10-27511-0.service\": RecentStats: unable to find data in memory cache]" Nov 04 01:26:42.747820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.747794 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:26:42.868457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868419 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-credential-socket\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.868637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868480 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-envoy\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.868637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868547 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b47cf65d-ee47-4303-baad-8911456e7299-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47cf65d-ee47-4303-baad-8911456e7299-istiod-ca-cert\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.868637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868585 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-socket\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.868814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868656 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-istio-token\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-istio-token\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.868814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868716 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-certs\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.868814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868714 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-credential-socket" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:42.868814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868753 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-kube-api-access-bwh8d\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-kube-api-access-bwh8d\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.868814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868804 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b47cf65d-ee47-4303-baad-8911456e7299-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47cf65d-ee47-4303-baad-8911456e7299-istio-podinfo\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.869091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868969 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47cf65d-ee47-4303-baad-8911456e7299-istiod-ca-cert" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:26:42.869091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868967 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-socket" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:42.869091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.868989 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-data\") pod \"b47cf65d-ee47-4303-baad-8911456e7299\" (UID: \"b47cf65d-ee47-4303-baad-8911456e7299\") " Nov 04 01:26:42.869284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.869243 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-data" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:42.869456 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.869434 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-certs" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:42.870166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.870140 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.870313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.870291 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.870313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.870314 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.870506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.870328 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47cf65d-ee47-4303-baad-8911456e7299-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.870506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.870342 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.871427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.871372 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-envoy" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:26:42.871776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.871742 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-istio-token" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:42.871860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.871816 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-kube-api-access-bwh8d" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "kube-api-access-bwh8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:26:42.871952 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.871936 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b47cf65d-ee47-4303-baad-8911456e7299-istio-podinfo" pod "b47cf65d-ee47-4303-baad-8911456e7299" (UID: "b47cf65d-ee47-4303-baad-8911456e7299"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:26:42.910272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.910243 5051 scope.go:122] "RemoveContainer" containerID="183f04992f9642fd7d2c12c4766698bd3f11ba6b728faf8b9235bd2aca007643" Nov 04 01:26:42.923281 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.923252 5051 scope.go:122] "RemoveContainer" containerID="cb0c41e104c9016085e708da78bffe0e38c63e0162921aa21e8cb6a5d332d2e3" Nov 04 01:26:42.934677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.934652 5051 scope.go:122] "RemoveContainer" containerID="60919e92ced66d2bc32d1015bf395c775030b588c6f9277e2a561588609bc223" Nov 04 01:26:42.946002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.945981 5051 scope.go:122] "RemoveContainer" containerID="5be96dd9cf80e39dde0821555c30c6c613d644b80c49d887ca43b3dc5f3b7aca" Nov 04 01:26:42.957463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.957444 5051 scope.go:122] "RemoveContainer" containerID="9c62173e4d00315b43ca73eeeb8df5f526a4e734090ffb48a97f68c3df01ebba" Nov 04 01:26:42.969814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.969795 5051 scope.go:122] "RemoveContainer" containerID="b4d124ea5b0ce52fd1c91b3f8cd4346febbeb5bfa75bb22b94603f038eb09ec3" Nov 04 01:26:42.971988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.971957 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47cf65d-ee47-4303-baad-8911456e7299-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.971988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.971984 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.972171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.972001 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwh8d\" (UniqueName: \"kubernetes.io/projected/b47cf65d-ee47-4303-baad-8911456e7299-kube-api-access-bwh8d\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.972171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.972019 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47cf65d-ee47-4303-baad-8911456e7299-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:26:42.972274 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.972184 5051 generic.go:356] "Generic (PLEG): container finished" podID="b47cf65d-ee47-4303-baad-8911456e7299" containerID="de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89" exitCode=0 Nov 04 01:26:42.972327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.972301 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" Nov 04 01:26:42.973074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.973044 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" event={"ID":"b47cf65d-ee47-4303-baad-8911456e7299","Type":"ContainerDied","Data":"de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89"} Nov 04 01:26:42.973208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.973158 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4" event={"ID":"b47cf65d-ee47-4303-baad-8911456e7299","Type":"ContainerDied","Data":"df495e63a14ca5e85ccecaf1791bd2cd2ef9389e8dbbb6bd283a5d98cf73558d"} Nov 04 01:26:42.973208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.973190 5051 scope.go:122] "RemoveContainer" containerID="de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89" Nov 04 01:26:42.988644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.988623 5051 scope.go:122] "RemoveContainer" containerID="9f0a9ad3489c637b2bb6fa948450f165473325afa233bfd43ed05fcbf95b9ebb" Nov 04 01:26:42.989204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.989170 5051 scope.go:122] "RemoveContainer" containerID="de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89" Nov 04 01:26:42.989504 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:26:42.989473 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89\": container with ID starting with de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89 not found: ID does not exist" containerID="de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89" Nov 04 01:26:42.989594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:42.989508 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89"} err="failed to get container status \"de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89\": rpc error: code = NotFound desc = could not find container \"de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89\": container with ID starting with de4b8ba7e8ccef6f81409a92cbb1adf39edb13ab8345d2270eeaca0d2bad4a89 not found: ID does not exist" Nov 04 01:26:43.001584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.001565 5051 scope.go:122] "RemoveContainer" containerID="7b9cab002a3e94a5a5f60c1da764dfdb270d7de998f5cddde3614bc2e39b0524" Nov 04 01:26:43.010112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.010085 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4"] Nov 04 01:26:43.016159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.016136 5051 scope.go:122] "RemoveContainer" containerID="4227997806ec4d0f397fc60800b40fc1e643629da9e41b9391bee033dcf1769f" Nov 04 01:26:43.018003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.017984 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-296c124f-openshift-default-687f659d99-xfwb4"] Nov 04 01:26:43.026901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.026881 5051 scope.go:122] "RemoveContainer" containerID="87574b9a5ee91fa05dc391ab3dd7a9c0b6b3668d51dc5ee9e18d5f513460d507" Nov 04 01:26:43.066195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.066165 5051 scope.go:122] "RemoveContainer" containerID="422d3ab04505d30171637b414afbe5bf51dc9f9d61724ccd47a4cd3a00848c5d" Nov 04 01:26:43.079230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.079205 5051 scope.go:122] "RemoveContainer" containerID="b29fe3d36b2abfb261d0c02ded7bfd88e01cbebb2f370c86cdada5dc4342bf18" Nov 04 01:26:43.093181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.093162 5051 scope.go:122] "RemoveContainer" containerID="4856d251165a9c75bfed0f1533c2ecc0e446b95d7225759afb0fe8bed6fc8e3a" Nov 04 01:26:43.104204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.104185 5051 scope.go:122] "RemoveContainer" containerID="ac1083756ab320e96fd9d47840914ce4fdcd09312ccafe4b22163b5331500968" Nov 04 01:26:43.114704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.114682 5051 scope.go:122] "RemoveContainer" containerID="0c2635431de4b23f0c604868916a762bd1e32090e960496d5869b9bcade94e2e" Nov 04 01:26:43.125871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.125848 5051 scope.go:122] "RemoveContainer" containerID="d841f907659d2433866e17c40d6861655f67cf5ddc8eb5b808583121f10d5a02" Nov 04 01:26:43.140532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.140513 5051 scope.go:122] "RemoveContainer" containerID="f56004a0e7a8c3a288cb6ce3c77ef7cc1a53c77b5de6cc60ab6ab9b6de04e8d4" Nov 04 01:26:43.192158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:26:43.192096 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47cf65d-ee47-4303-baad-8911456e7299" path="/var/lib/kubelet/pods/b47cf65d-ee47-4303-baad-8911456e7299/volumes" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.562278 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6"] Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563455 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0d53f41f-fb95-473f-981f-a0752c78dd8d" containerName="istio-proxy" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563478 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d53f41f-fb95-473f-981f-a0752c78dd8d" containerName="istio-proxy" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563501 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b47cf65d-ee47-4303-baad-8911456e7299" containerName="istio-proxy" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563508 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47cf65d-ee47-4303-baad-8911456e7299" containerName="istio-proxy" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563520 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c72b5492-f761-46d0-aa80-2c9abb6b0b81" containerName="revoke-keys" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563529 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72b5492-f761-46d0-aa80-2c9abb6b0b81" containerName="revoke-keys" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563765 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d53f41f-fb95-473f-981f-a0752c78dd8d" containerName="istio-proxy" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563785 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47cf65d-ee47-4303-baad-8911456e7299" containerName="istio-proxy" Nov 04 01:27:00.571141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.563793 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72b5492-f761-46d0-aa80-2c9abb6b0b81" containerName="revoke-keys" Nov 04 01:27:00.596006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.595977 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6"] Nov 04 01:27:00.596283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.596033 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.605341 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.605311 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-ef6cc9-openshift-default-dockercfg-m7xbv\"" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.748918 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.749064 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.749134 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.749224 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.749414 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drcgq\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-kube-api-access-drcgq\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.749524 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.749780 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.749816 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.751143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.749854 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.892199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.865761 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.892199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.865844 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.892199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.865874 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.892199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.865907 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.892199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.865995 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drcgq\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-kube-api-access-drcgq\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.892199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.866085 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.908162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.902210 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.928144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.922483 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.928144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.922536 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.928144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.922574 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.928144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.923042 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.928144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.923730 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.928144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.923972 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:00.928144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:00.927329 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:01.055146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.054448 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:01.100146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.092059 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:01.117164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.113475 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drcgq\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-kube-api-access-drcgq\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:01.117164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.113984 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:01.218421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.218341 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:01.584728 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:27:01.584655 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747c4b7d_4c96_42ab_b459_8eb5aaf75e09.slice/crio-765194c514c2e5d8a73d9174772fffa7062c1317e32cedc504fd1995ac86e257 WatchSource:0}: Error finding container 765194c514c2e5d8a73d9174772fffa7062c1317e32cedc504fd1995ac86e257: Status 404 returned error can't find the container with id 765194c514c2e5d8a73d9174772fffa7062c1317e32cedc504fd1995ac86e257 Nov 04 01:27:01.585425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.585370 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6"] Nov 04 01:27:01.587843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.587800 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:27:01.587933 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.587891 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:27:01.587933 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:01.587928 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:27:02.321857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:02.321817 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" event={"ID":"747c4b7d-4c96-42ab-b459-8eb5aaf75e09","Type":"ContainerStarted","Data":"5170c75c442c95a9d06c312d180443dcf6dec2ec9e051c9b36e05411e4c910fc"} Nov 04 01:27:02.322065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:02.321875 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" event={"ID":"747c4b7d-4c96-42ab-b459-8eb5aaf75e09","Type":"ContainerStarted","Data":"765194c514c2e5d8a73d9174772fffa7062c1317e32cedc504fd1995ac86e257"} Nov 04 01:27:02.346711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:02.346658 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" podStartSLOduration=2.346643696 podStartE2EDuration="2.346643696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:02.340823462 +0000 UTC m=+1761.312572363" watchObservedRunningTime="2026-11-04 01:27:02.346643696 +0000 UTC m=+1761.318392557" Nov 04 01:27:03.230481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:03.230397 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:03.239220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:03.239194 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:03.330358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:03.330327 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:03.331169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:03.331104 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:27:04.067148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:04.063174 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd"] Nov 04 01:27:04.120739 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:04.120389 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-688f431e-cmjkd"] Nov 04 01:27:04.160108 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:27:04.160070 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:27:05.192500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:05.192465 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f4aab3-be13-4af0-8b01-25843096202d" path="/var/lib/kubelet/pods/55f4aab3-be13-4af0-8b01-25843096202d/volumes" Nov 04 01:27:06.027560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.027524 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl"] Nov 04 01:27:06.032633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.032606 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:06.040078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.040054 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl"] Nov 04 01:27:06.123145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.110506 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-ef6cc9\"" Nov 04 01:27:06.135143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.128193 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-ef6cc9-dockercfg-5hvdx\"" Nov 04 01:27:06.182143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.174488 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwxg\" (UniqueName: \"kubernetes.io/projected/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-kube-api-access-bzwxg\") pod \"payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl\" (UID: \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:06.182143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.174672 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl\" (UID: \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:06.479498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.455333 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl\" (UID: \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:06.479498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.455703 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwxg\" (UniqueName: \"kubernetes.io/projected/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-kube-api-access-bzwxg\") pod \"payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl\" (UID: \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:06.494141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.485345 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl\" (UID: \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:06.494141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.490341 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwxg\" (UniqueName: \"kubernetes.io/projected/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-kube-api-access-bzwxg\") pod \"payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl\" (UID: \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:06.667922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:06.667885 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:07.138139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:07.132243 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl"] Nov 04 01:27:07.475488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:07.475320 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" event={"ID":"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d","Type":"ContainerStarted","Data":"b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e"} Nov 04 01:27:07.475488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:07.475397 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" event={"ID":"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d","Type":"ContainerStarted","Data":"638117b962b243eb501681612d2582265d4cc0cde937d4e622b0e2927004da6c"} Nov 04 01:27:07.475778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:07.475756 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:07.509344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:07.509298 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" podStartSLOduration=2.509284784 podStartE2EDuration="2.509284784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:07.505945611 +0000 UTC m=+1766.477694492" watchObservedRunningTime="2026-11-04 01:27:07.509284784 +0000 UTC m=+1766.481033654" Nov 04 01:27:09.079604 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:09.079238 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn"] Nov 04 01:27:09.087178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:09.087145 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bc2f54f-dtsvn"] Nov 04 01:27:09.190441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:09.190410 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cef703d-c353-45d3-94f7-b0f5a460e27a" path="/var/lib/kubelet/pods/3cef703d-c353-45d3-94f7-b0f5a460e27a/volumes" Nov 04 01:27:13.076968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:13.076934 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc"] Nov 04 01:27:13.082040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:13.082013 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0370b5c7-vswkc"] Nov 04 01:27:13.205498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:13.205465 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb8a4ec-5f08-48f2-98cd-865a6face69a" path="/var/lib/kubelet/pods/7fb8a4ec-5f08-48f2-98cd-865a6face69a/volumes" Nov 04 01:27:18.492988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:18.492953 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:27:22.471754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.471708 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486"] Nov 04 01:27:22.509707 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.509663 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486"] Nov 04 01:27:22.509896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.509729 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2"] Nov 04 01:27:22.510137 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.510090 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:22.517996 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.517957 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:22.523333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.523310 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3888c434\"" Nov 04 01:27:22.523625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.523607 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3888c434-dockercfg-pdcwf\"" Nov 04 01:27:22.525863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.525836 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2"] Nov 04 01:27:22.761271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.743112 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkm49\" (UniqueName: \"kubernetes.io/projected/5008b044-5916-4244-9e18-a1478be5e552-kube-api-access-fkm49\") pod \"payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486\" (UID: \"5008b044-5916-4244-9e18-a1478be5e552\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:22.761271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.743214 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt895\" (UniqueName: \"kubernetes.io/projected/a782226f-9d6a-4948-979d-d8efc336678a-kube-api-access-gt895\") pod \"payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2\" (UID: \"a782226f-9d6a-4948-979d-d8efc336678a\") " pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:22.761271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.743312 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5008b044-5916-4244-9e18-a1478be5e552-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486\" (UID: \"5008b044-5916-4244-9e18-a1478be5e552\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:22.761271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:22.743373 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782226f-9d6a-4948-979d-d8efc336678a-plugins-config-volume\") pod \"payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2\" (UID: \"a782226f-9d6a-4948-979d-d8efc336678a\") " pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:23.032194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.021105 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkm49\" (UniqueName: \"kubernetes.io/projected/5008b044-5916-4244-9e18-a1478be5e552-kube-api-access-fkm49\") pod \"payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486\" (UID: \"5008b044-5916-4244-9e18-a1478be5e552\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:23.063060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.021275 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt895\" (UniqueName: \"kubernetes.io/projected/a782226f-9d6a-4948-979d-d8efc336678a-kube-api-access-gt895\") pod \"payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2\" (UID: \"a782226f-9d6a-4948-979d-d8efc336678a\") " pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:23.063280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.063250 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5008b044-5916-4244-9e18-a1478be5e552-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486\" (UID: \"5008b044-5916-4244-9e18-a1478be5e552\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:23.063430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.063394 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782226f-9d6a-4948-979d-d8efc336678a-plugins-config-volume\") pod \"payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2\" (UID: \"a782226f-9d6a-4948-979d-d8efc336678a\") " pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:23.064179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.064153 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5008b044-5916-4244-9e18-a1478be5e552-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486\" (UID: \"5008b044-5916-4244-9e18-a1478be5e552\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:23.067655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.067633 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkm49\" (UniqueName: \"kubernetes.io/projected/5008b044-5916-4244-9e18-a1478be5e552-kube-api-access-fkm49\") pod \"payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486\" (UID: \"5008b044-5916-4244-9e18-a1478be5e552\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:23.077528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.077499 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782226f-9d6a-4948-979d-d8efc336678a-plugins-config-volume\") pod \"payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2\" (UID: \"a782226f-9d6a-4948-979d-d8efc336678a\") " pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:23.210130 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.208818 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:23.233229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.212358 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt895\" (UniqueName: \"kubernetes.io/projected/a782226f-9d6a-4948-979d-d8efc336678a-kube-api-access-gt895\") pod \"payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2\" (UID: \"a782226f-9d6a-4948-979d-d8efc336678a\") " pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:23.365315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.365275 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:23.472846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.472807 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9"] Nov 04 01:27:23.554796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.554757 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e335f439-bv6l9"] Nov 04 01:27:23.939303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.939268 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486"] Nov 04 01:27:23.940759 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:27:23.940527 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5008b044_5916_4244_9e18_a1478be5e552.slice/crio-8ecd3b2bd50776363519cfe4a9355e3a514ee4a3683b04b0ba6d090d6c0c0764 WatchSource:0}: Error finding container 8ecd3b2bd50776363519cfe4a9355e3a514ee4a3683b04b0ba6d090d6c0c0764: Status 404 returned error can't find the container with id 8ecd3b2bd50776363519cfe4a9355e3a514ee4a3683b04b0ba6d090d6c0c0764 Nov 04 01:27:23.971482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:23.971415 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2"] Nov 04 01:27:24.073626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:24.073588 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" event={"ID":"a782226f-9d6a-4948-979d-d8efc336678a","Type":"ContainerStarted","Data":"c98a67b442828e74f5cb9261328bfde98532daed193bbc10e7d63de45715f0ec"} Nov 04 01:27:24.073798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:24.073654 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" event={"ID":"a782226f-9d6a-4948-979d-d8efc336678a","Type":"ContainerStarted","Data":"dad37c4d312e7d5a3642ca4a6cfb782ef99063ba6e5b24abe8779471698b8766"} Nov 04 01:27:24.076154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:24.074799 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:24.079053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:24.077556 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" event={"ID":"5008b044-5916-4244-9e18-a1478be5e552","Type":"ContainerStarted","Data":"f806114c71ad76dcee76b0f8e8eed71119456f326833fe309a6297247f938955"} Nov 04 01:27:24.079053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:24.077610 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" event={"ID":"5008b044-5916-4244-9e18-a1478be5e552","Type":"ContainerStarted","Data":"8ecd3b2bd50776363519cfe4a9355e3a514ee4a3683b04b0ba6d090d6c0c0764"} Nov 04 01:27:24.079053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:24.078407 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:24.098545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:24.098502 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" podStartSLOduration=2.098489196 podStartE2EDuration="2.098489196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:24.094284736 +0000 UTC m=+1783.066033608" watchObservedRunningTime="2026-11-04 01:27:24.098489196 +0000 UTC m=+1783.070238083" Nov 04 01:27:24.124079 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:24.124029 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" podStartSLOduration=2.124014735 podStartE2EDuration="2.124014735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:24.121005918 +0000 UTC m=+1783.092754782" watchObservedRunningTime="2026-11-04 01:27:24.124014735 +0000 UTC m=+1783.095763605" Nov 04 01:27:25.226140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:25.224219 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd49052-d139-4365-b55c-9ea489e4922b" path="/var/lib/kubelet/pods/5dd49052-d139-4365-b55c-9ea489e4922b/volumes" Nov 04 01:27:27.977145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:27.968909 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.262140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:28.259217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.505963 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:28.496837 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:28.542210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:28.523909 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:29.367316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:29.367252 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:27:29.386000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:29.367920 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:27:29.528041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:29.527906 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:27:29.866292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:29.866255 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:27:29.883290 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:29.883258 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:27:30.030314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:30.030282 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:27:30.058277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:30.058243 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:27:33.975778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:33.975731 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4"] Nov 04 01:27:33.991451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:33.991415 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.007140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:33.998500 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:27:34.007140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:33.998638 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:27:34.007140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:33.998725 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:27:34.007140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.002772 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:27:34.052140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.033385 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4"] Nov 04 01:27:34.073776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.073183 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msl8k\" (UniqueName: \"kubernetes.io/projected/3447fade-e00b-48a5-823a-2ace4656486f-kube-api-access-msl8k\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.073776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.073263 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.073776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.073498 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.189216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.184339 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.189216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.184555 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msl8k\" (UniqueName: \"kubernetes.io/projected/3447fade-e00b-48a5-823a-2ace4656486f-kube-api-access-msl8k\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.189216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.184617 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.190266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.190238 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.220150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.220042 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msl8k\" (UniqueName: \"kubernetes.io/projected/3447fade-e00b-48a5-823a-2ace4656486f-kube-api-access-msl8k\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.226114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.226031 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46bef393-m22c4\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.399174 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:27:34.370444 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:27:34.405726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.405427 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:34.827985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:34.825551 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4"] Nov 04 01:27:35.474009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:35.473913 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" event={"ID":"3447fade-e00b-48a5-823a-2ace4656486f","Type":"ContainerStarted","Data":"1d24e0f38f51239b98eba6e943827a2ef283bb5e3495aa010d6513e9e62f580c"} Nov 04 01:27:35.474009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:35.473980 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" event={"ID":"3447fade-e00b-48a5-823a-2ace4656486f","Type":"ContainerStarted","Data":"29d01ae832fbd8ac6fb9e6d29b0e43aeab89a70047f108cc50c3ec79ec9a3974"} Nov 04 01:27:35.498414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:35.498356 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" podStartSLOduration=2.498338022 podStartE2EDuration="2.498338022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:35.496482018 +0000 UTC m=+1794.468230890" watchObservedRunningTime="2026-11-04 01:27:35.498338022 +0000 UTC m=+1794.470086894" Nov 04 01:27:36.312831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:36.311557 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:27:36.312831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:36.312143 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:27:36.508964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:36.508928 5051 generic.go:356] "Generic (PLEG): container finished" podID="3447fade-e00b-48a5-823a-2ace4656486f" containerID="1d24e0f38f51239b98eba6e943827a2ef283bb5e3495aa010d6513e9e62f580c" exitCode=0 Nov 04 01:27:36.509383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:36.508979 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" event={"ID":"3447fade-e00b-48a5-823a-2ace4656486f","Type":"ContainerDied","Data":"1d24e0f38f51239b98eba6e943827a2ef283bb5e3495aa010d6513e9e62f580c"} Nov 04 01:27:37.853004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:37.852972 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:37.944148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:37.944100 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3447fade-e00b-48a5-823a-2ace4656486f-kube-api-access-msl8k\" (UniqueName: \"kubernetes.io/projected/3447fade-e00b-48a5-823a-2ace4656486f-kube-api-access-msl8k\") pod \"3447fade-e00b-48a5-823a-2ace4656486f\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " Nov 04 01:27:37.944334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:37.944223 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-xks-ca\") pod \"3447fade-e00b-48a5-823a-2ace4656486f\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " Nov 04 01:27:37.944334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:37.944328 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-openshift-ca\") pod \"3447fade-e00b-48a5-823a-2ace4656486f\" (UID: \"3447fade-e00b-48a5-823a-2ace4656486f\") " Nov 04 01:27:37.945108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:37.945079 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-openshift-ca" pod "3447fade-e00b-48a5-823a-2ace4656486f" (UID: "3447fade-e00b-48a5-823a-2ace4656486f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:27:37.947537 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:37.947513 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-xks-ca" pod "3447fade-e00b-48a5-823a-2ace4656486f" (UID: "3447fade-e00b-48a5-823a-2ace4656486f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:27:37.947901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:37.947883 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3447fade-e00b-48a5-823a-2ace4656486f-kube-api-access-msl8k" pod "3447fade-e00b-48a5-823a-2ace4656486f" (UID: "3447fade-e00b-48a5-823a-2ace4656486f"). InnerVolumeSpecName "kube-api-access-msl8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:27:38.029083 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:38.029038 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2"] Nov 04 01:27:38.035416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:38.035381 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccedf22b-qdqp2"] Nov 04 01:27:38.046518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:38.046486 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:38.046518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:38.046517 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msl8k\" (UniqueName: \"kubernetes.io/projected/3447fade-e00b-48a5-823a-2ace4656486f-kube-api-access-msl8k\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:38.046708 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:38.046534 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3447fade-e00b-48a5-823a-2ace4656486f-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:27:38.531961 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:38.531921 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" event={"ID":"3447fade-e00b-48a5-823a-2ace4656486f","Type":"ContainerDied","Data":"29d01ae832fbd8ac6fb9e6d29b0e43aeab89a70047f108cc50c3ec79ec9a3974"} Nov 04 01:27:38.532181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:38.531991 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29d01ae832fbd8ac6fb9e6d29b0e43aeab89a70047f108cc50c3ec79ec9a3974" Nov 04 01:27:38.532181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:38.532066 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4" Nov 04 01:27:39.195705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:39.195664 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c3b55d-f7da-4811-80f3-7b8451a6d3d2" path="/var/lib/kubelet/pods/11c3b55d-f7da-4811-80f3-7b8451a6d3d2/volumes" Nov 04 01:27:40.731821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.731781 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb"] Nov 04 01:27:40.732936 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.732911 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3447fade-e00b-48a5-823a-2ace4656486f" containerName="revoke-keys" Nov 04 01:27:40.733012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.732940 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3447fade-e00b-48a5-823a-2ace4656486f" containerName="revoke-keys" Nov 04 01:27:40.733165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.733151 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3447fade-e00b-48a5-823a-2ace4656486f" containerName="revoke-keys" Nov 04 01:27:40.738278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.738256 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:40.753267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.753236 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2fff32\"" Nov 04 01:27:40.753420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.753268 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2fff32-dockercfg-gl2n5\"" Nov 04 01:27:40.773553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.773493 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb"] Nov 04 01:27:40.907146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.900240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba88bc8f-54a5-4b80-a545-343ecbc95551-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb\" (UID: \"ba88bc8f-54a5-4b80-a545-343ecbc95551\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:40.907146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:40.900372 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnvcd\" (UniqueName: \"kubernetes.io/projected/ba88bc8f-54a5-4b80-a545-343ecbc95551-kube-api-access-pnvcd\") pod \"payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb\" (UID: \"ba88bc8f-54a5-4b80-a545-343ecbc95551\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:41.097146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.072631 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba88bc8f-54a5-4b80-a545-343ecbc95551-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb\" (UID: \"ba88bc8f-54a5-4b80-a545-343ecbc95551\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:41.103572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.103529 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnvcd\" (UniqueName: \"kubernetes.io/projected/ba88bc8f-54a5-4b80-a545-343ecbc95551-kube-api-access-pnvcd\") pod \"payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb\" (UID: \"ba88bc8f-54a5-4b80-a545-343ecbc95551\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:41.104431 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.074013 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba88bc8f-54a5-4b80-a545-343ecbc95551-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb\" (UID: \"ba88bc8f-54a5-4b80-a545-343ecbc95551\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:41.172517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.172481 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnvcd\" (UniqueName: \"kubernetes.io/projected/ba88bc8f-54a5-4b80-a545-343ecbc95551-kube-api-access-pnvcd\") pod \"payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb\" (UID: \"ba88bc8f-54a5-4b80-a545-343ecbc95551\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:41.360858 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.360773 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2fff32-dockercfg-gl2n5\"" Nov 04 01:27:41.368903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.368864 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:41.616131 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.616016 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:27:41.724970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.724936 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:27:41.871370 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:27:41.871337 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba88bc8f_54a5_4b80_a545_343ecbc95551.slice/crio-7edef1a264265bce9649f0967f34e105d4daea8cbe29cfc9acc2c95add4d32eb WatchSource:0}: Error finding container 7edef1a264265bce9649f0967f34e105d4daea8cbe29cfc9acc2c95add4d32eb: Status 404 returned error can't find the container with id 7edef1a264265bce9649f0967f34e105d4daea8cbe29cfc9acc2c95add4d32eb Nov 04 01:27:41.874698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:41.874669 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb"] Nov 04 01:27:42.713866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:42.713826 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" event={"ID":"ba88bc8f-54a5-4b80-a545-343ecbc95551","Type":"ContainerStarted","Data":"6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16"} Nov 04 01:27:42.714076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:42.713894 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" event={"ID":"ba88bc8f-54a5-4b80-a545-343ecbc95551","Type":"ContainerStarted","Data":"7edef1a264265bce9649f0967f34e105d4daea8cbe29cfc9acc2c95add4d32eb"} Nov 04 01:27:42.714076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:42.713922 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:27:42.734303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:42.734246 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" podStartSLOduration=2.734233142 podStartE2EDuration="2.734233142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:27:42.731922257 +0000 UTC m=+1801.703671130" watchObservedRunningTime="2026-11-04 01:27:42.734233142 +0000 UTC m=+1801.705982013" Nov 04 01:27:43.474624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:43.474587 5051 scope.go:122] "RemoveContainer" containerID="1fa0c94f72e69f2edcbc3b2e162d94421abf366c368dfbef77bb5b54505df398" Nov 04 01:27:43.487653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:43.487625 5051 scope.go:122] "RemoveContainer" containerID="4d7e242da2bf3a9e4a1ba2902feda5690f596d3786bcb345ca4910d587a033f1" Nov 04 01:27:43.499685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:43.499661 5051 scope.go:122] "RemoveContainer" containerID="7dfc10ccf4dbd7f65c4ca2dfd3f1662086b3bf9e4e3fce5dcfb960b5253b40b7" Nov 04 01:27:43.561217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:43.558606 5051 scope.go:122] "RemoveContainer" containerID="828035fc1eebbdf0d0d5e4d0c1845f4120c049cdf8fd0bf40620727ebca65291" Nov 04 01:27:43.763147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:43.752795 5051 scope.go:122] "RemoveContainer" containerID="75fd2fd5b58c24b90e4d9c6a0a54e517f0f7e841dbdfab89a2046e5003acb736" Nov 04 01:27:44.012294 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:44.011991 5051 scope.go:122] "RemoveContainer" containerID="38b2f08d5df526774f1672d8ee92674323c61716ab8c7c4444fe6200924244a6" Nov 04 01:27:44.102685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:44.101714 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv"] Nov 04 01:27:44.118510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:44.118482 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d3e62d0-b8dhv"] Nov 04 01:27:45.335636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:45.335580 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07cccfa6-abc9-44a4-9a84-762a19956113" path="/var/lib/kubelet/pods/07cccfa6-abc9-44a4-9a84-762a19956113/volumes" Nov 04 01:27:46.029715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:46.029680 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt"] Nov 04 01:27:46.038034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:46.038003 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a40b686-96gkt"] Nov 04 01:27:47.209962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:47.209845 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d72759-d28a-4941-a1dc-69c107b65b40" path="/var/lib/kubelet/pods/e9d72759-d28a-4941-a1dc-69c107b65b40/volumes" Nov 04 01:27:53.907891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:53.907856 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:28:00.252813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:27:59.498049 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" podUID="59ac90c2-5b91-45fb-a1b2-b443b86668d8" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:00.720799 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh"] Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:01.467311 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:01.008233 5051 patch_prober.go:29] interesting pod/jobset-controller-manager-69d5854bb-87mwv container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:01.722985 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" podUID="9d5e6ac7-97c0-407d-b050-bf4d8670b7e9" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.419676 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb153fc-0c8d-4625-8b59-36344e1f3766-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.419832 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.419993 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.420145 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4bq9\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-kube-api-access-f4bq9\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.420202 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.420249 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.420355 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.420416 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.420554 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.522795 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.522860 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4bq9\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-kube-api-access-f4bq9\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.522884 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.522909 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.522955 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.522980 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.523031 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.523244 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb153fc-0c8d-4625-8b59-36344e1f3766-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.523278 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.523401 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.523450 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.523523 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.523633 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:02.527367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:02.524071 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb153fc-0c8d-4625-8b59-36344e1f3766-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:03.629650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:03.545662 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:03.629650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:03.546395 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:03.629650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:03.546706 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4bq9\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-kube-api-access-f4bq9\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:03.629650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:03.546861 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:07.124521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:04.594298 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:07.124521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:04.795230 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:07.124521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:04.827871 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:28:07.124521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:04.907282 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.540102 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.540242 5051 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-7wfdb container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.30:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.540332 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-7wfdb" podUID="112a671a-1e5f-4fe3-bc77-deb979fc4566" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.30:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.542041 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.571513 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-j49c7" podUID="1006149a-92ee-43c1-bb41-6aaf1df41c36" containerName="sbdb" probeResult="failure" output="command timed out" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.571798 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-wtpbl" podUID="e7f2ccca-45df-41f8-bbf9-0b8aaa791160" containerName="extractor" probeResult="failure" output="command timed out" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.587222 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.587552 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.592331 5051 patch_prober.go:29] interesting pod/dns-default-mvtzk container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.20:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.592364 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-mvtzk" podUID="b0807c2d-b756-42e8-86b3-97c30f23df72" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.20:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.593017 5051 trace.go:236] Trace[795512415]: "Calculate volume metrics of kube-api-access-6zjmd for pod kuadrant-system/authorino-7457bc8fc9-m4sb2" (04-Nov-2026 01:28:28.455) (total time: 1137ms): Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: Trace[795512415]: [1.137180572s] [1.137180572s] END Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.595418 5051 trace.go:236] Trace[1394212644]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" (04-Nov-2026 01:28:28.590) (total time: 1005ms): Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: Trace[1394212644]: [1.005232706s] [1.005232706s] END Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.599828 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" podUID="ba88bc8f-54a5-4b80-a545-343ecbc95551" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.160:50052\" within 1s: context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:28:29.601230 5051 logging.go:55] [core] [Channel #73 SubChannel #75]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.160:50052", ServerName: "10.132.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.160:50052: operation was canceled" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.604792 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.604828 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.610023 5051 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.623835696s: [/var/lib/containers/storage/overlay/c535690ee4cb29c67aa1edbe865ad8ef78e30e2520d26780a9a8b6c1a8aad62a/diff /var/log/pods/openshift-ingress_e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57_60599da2-18ec-4ac3-8968-e6726cb5d369/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.642375 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.667575 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.669517 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" podUID="f80b3f88-24db-4e50-aecc-c1c19664e0ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.695624 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.696641 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.704542 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.859315 5051 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-cpht7 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": context deadline exceeded" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.859402 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" podUID="2da3265c-ae95-4220-a9d0-11ab44432db6" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.881239 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" podUID="59ac90c2-5b91-45fb-a1b2-b443b86668d8" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.954572 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.955165 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.966700 5051 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-cpht7 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": context deadline exceeded" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.966763 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" podUID="2da3265c-ae95-4220-a9d0-11ab44432db6" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.967688 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:29.967826 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.001404 5051 patch_prober.go:29] interesting pod/router-default-669749fcf7-vqftg container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.15:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.001476 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-669749fcf7-vqftg" podUID="7d024224-166d-4cb4-976f-75e0b47addae" containerName="router" probeResult="failure" output="Get \"http://10.132.0.15:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.031428 5051 patch_prober.go:29] interesting pod/jobset-controller-manager-69d5854bb-87mwv container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.031493 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" podUID="9d5e6ac7-97c0-407d-b050-bf4d8670b7e9" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.077868 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.181217 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.182226 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.182281 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.541335 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.542223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.542277 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" podUID="7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.156:50052\" within 1s: context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.544823 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.549438 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.560579 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.562481 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:28:30.583282 5051 logging.go:55] [core] [Channel #74 SubChannel #79]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.156:50052", ServerName: "10.132.0.156:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.132.0.2:47246->10.132.0.156:50052: use of closed network connection" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.588354 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.588368 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.595619 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.598223 5051 patch_prober.go:29] interesting pod/router-default-669749fcf7-vqftg container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.15:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.598256 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-669749fcf7-vqftg" podUID="7d024224-166d-4cb4-976f-75e0b47addae" containerName="router" probeResult="failure" output="Get \"http://10.132.0.15:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.617396 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.617483 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.629339 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" podUID="59ac90c2-5b91-45fb-a1b2-b443b86668d8" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.635195 5051 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-7vtl7 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.18:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.635261 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.643518 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.644623 5051 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-7vtl7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.644678 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.657267 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" podUID="ba88bc8f-54a5-4b80-a545-343ecbc95551" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.681256 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.696466 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.696968 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.703532 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.712369 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.713387 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.740241 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" podUID="c9f5857b-b55b-4002-b4a3-4f61950bd6ba" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.747234 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" podUID="c9f5857b-b55b-4002-b4a3-4f61950bd6ba" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.761295 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.761820 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.761946 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.847476 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.859685 5051 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-cpht7 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.859741 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" podUID="2da3265c-ae95-4220-a9d0-11ab44432db6" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.864878 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.923238 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk" podUID="59ac90c2-5b91-45fb-a1b2-b443b86668d8" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.955493 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.955509 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.967782 5051 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-cpht7 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.967828 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cpht7" podUID="2da3265c-ae95-4220-a9d0-11ab44432db6" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:30.967979 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.043277 5051 patch_prober.go:29] interesting pod/router-default-669749fcf7-vqftg container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.15:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.043330 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-669749fcf7-vqftg" podUID="7d024224-166d-4cb4-976f-75e0b47addae" containerName="router" probeResult="failure" output="Get \"http://10.132.0.15:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.073236 5051 patch_prober.go:29] interesting pod/jobset-controller-manager-69d5854bb-87mwv container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.073303 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" podUID="9d5e6ac7-97c0-407d-b050-bf4d8670b7e9" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.182163 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.522241 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.542234 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.543186 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.545637 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.550321 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.561471 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.562615 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.589068 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.589602 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.636037 5051 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-7vtl7 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.18:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.636156 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.639300 5051 patch_prober.go:29] interesting pod/router-default-669749fcf7-vqftg container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.15:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.639349 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-669749fcf7-vqftg" podUID="7d024224-166d-4cb4-976f-75e0b47addae" containerName="router" probeResult="failure" output="Get \"http://10.132.0.15:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.644485 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.644984 5051 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-7vtl7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.645542 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.658263 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.658330 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.682383 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.697528 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.699680 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.712927 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.714025 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.782282 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" podUID="c9f5857b-b55b-4002-b4a3-4f61950bd6ba" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.788236 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" podUID="c9f5857b-b55b-4002-b4a3-4f61950bd6ba" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.802231 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.848472 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.865928 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.877852 5051 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6jrth container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.877907 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" podUID="47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:28:31.948834 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.118s" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:31.948869 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc"] Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.103362 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6"] Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.103472 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.228606 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.228704 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.228745 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.228890 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54pc\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-kube-api-access-n54pc\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.228992 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.229062 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.229145 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.229185 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.229300 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.319457 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.329481 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df6da4c0-openshift-default-dockercfg-78n5r\"" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.369597 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.369653 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:35.978179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.369784 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54pc\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-kube-api-access-n54pc\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.369860 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.369909 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.369959 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.369998 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.370102 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.370712 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472365 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472488 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvph\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-kube-api-access-6qvph\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472552 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472686 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472735 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472752 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c36e46-07ec-47df-a65c-d35233518e01-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472877 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472901 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c36e46-07ec-47df-a65c-d35233518e01-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.472926 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.492347 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.492379 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh"] Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574551 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574645 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvph\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-kube-api-access-6qvph\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574697 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574792 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574844 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574862 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c36e46-07ec-47df-a65c-d35233518e01-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574905 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574920 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c36e46-07ec-47df-a65c-d35233518e01-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.574953 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.676216 5051 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6jrth container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:32.686950 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" podUID="47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.727970 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh"] Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.728097 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.728240 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.746665 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.746734 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.746770 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.746808 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.747061 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.747113 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.747229 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.747445 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.747504 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.747541 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.747609 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752383 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752737 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752761 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752787 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752807 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752821 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752836 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752870 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752884 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.752897 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.773819 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-75f7ae-openshift-default-dockercfg-v4pjw\"" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.907034 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvdm\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-kube-api-access-qvvdm\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.907111 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.907202 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.907260 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.907609 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.908233 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.908938 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.909322 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:33.909790 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.012023 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.012307 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvdm\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-kube-api-access-qvvdm\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.012365 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.012519 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.012559 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.012852 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.013110 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.013546 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.013610 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.097782 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istiod-ca-cert\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.372238 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.372298 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.752829 5051 patch_prober.go:29] interesting pod/jobset-controller-manager-69d5854bb-87mwv container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.752888 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" podUID="9d5e6ac7-97c0-407d-b050-bf4d8670b7e9" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.43:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.829935 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/readyz\": read tcp 10.132.0.2:38054->10.132.0.44:8081: read: connection reset by peer" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.830166 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.831091 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/readyz\": dial tcp 10.132.0.44:8081: connect: connection refused" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.918199 5051 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6jrth container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:34.918248 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" podUID="47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:35.071698 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-data\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:35.071698 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:35.420325 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:35.555333 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-certs\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:35.729203 5051 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6jrth container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:35.729254 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" podUID="47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:35.923861 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-credential-socket\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:35.995479 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.069877 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.188608 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.336387 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c36e46-07ec-47df-a65c-d35233518e01-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.341417 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.341416 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.399072 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.485072 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.555560 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.944269 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.944588 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.944701 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:36.944837 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvdm\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-kube-api-access-qvvdm\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.139337 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-token\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.139394 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54pc\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-kube-api-access-n54pc\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.351418 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.351732 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.351756 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c36e46-07ec-47df-a65c-d35233518e01-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.351768 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvph\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-kube-api-access-6qvph\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.578882 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.664140 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-envoy\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.664440 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-podinfo\") pod \"e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.739946 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:37.920081 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:38.124192 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6"] Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:38.139010 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc"] Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:38.148570 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh"] Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:28:38.943470 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:38.999731 5051 generic.go:356] "Generic (PLEG): container finished" podID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerID="d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3" exitCode=0 Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:38.999803 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" event={"ID":"5c49724a-9d0f-42ec-86c8-cf8f7229f38a","Type":"ContainerDied","Data":"d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3"} Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.000416 5051 scope.go:122] "RemoveContainer" containerID="d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.002622 5051 generic.go:356] "Generic (PLEG): container finished" podID="c9f5857b-b55b-4002-b4a3-4f61950bd6ba" containerID="85b554c54591a77482b2b1e104692b9fac16a5fe0b6f6bf36c1043c5a770f690" exitCode=1 Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.002690 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" event={"ID":"c9f5857b-b55b-4002-b4a3-4f61950bd6ba","Type":"ContainerDied","Data":"85b554c54591a77482b2b1e104692b9fac16a5fe0b6f6bf36c1043c5a770f690"} Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.003244 5051 scope.go:122] "RemoveContainer" containerID="85b554c54591a77482b2b1e104692b9fac16a5fe0b6f6bf36c1043c5a770f690" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.058143 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k"] Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.140438 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.263352 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k"] Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.280623 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.291320 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-98f6f6-dockercfg-frf9c\"" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.302702 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-98f6f6\"" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.419832 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c4ea51-856e-4992-bc91-d291fa962765-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k\" (UID: \"77c4ea51-856e-4992-bc91-d291fa962765\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.420318 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bcm\" (UniqueName: \"kubernetes.io/projected/77c4ea51-856e-4992-bc91-d291fa962765-kube-api-access-f7bcm\") pod \"payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k\" (UID: \"77c4ea51-856e-4992-bc91-d291fa962765\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.525854 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c4ea51-856e-4992-bc91-d291fa962765-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k\" (UID: \"77c4ea51-856e-4992-bc91-d291fa962765\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.526190 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bcm\" (UniqueName: \"kubernetes.io/projected/77c4ea51-856e-4992-bc91-d291fa962765-kube-api-access-f7bcm\") pod \"payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k\" (UID: \"77c4ea51-856e-4992-bc91-d291fa962765\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.527583 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c4ea51-856e-4992-bc91-d291fa962765-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k\" (UID: \"77c4ea51-856e-4992-bc91-d291fa962765\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.611943 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bcm\" (UniqueName: \"kubernetes.io/projected/77c4ea51-856e-4992-bc91-d291fa962765-kube-api-access-f7bcm\") pod \"payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k\" (UID: \"77c4ea51-856e-4992-bc91-d291fa962765\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.750600 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:28:39.809666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:39.776523 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:40.868253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:40.783309 5051 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6jrth container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:40.868253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:40.783359 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jrth" podUID="47a1b8a9-e616-4cb0-9e1e-ce3ff75251f7" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:42.099197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:42.099164 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn"] Nov 04 01:28:42.104019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:42.103986 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6"] Nov 04 01:28:42.110442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:42.110419 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b521bb-phdbn"] Nov 04 01:28:42.115056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:42.115033 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38ec51f-2jvt6"] Nov 04 01:28:42.149202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:42.149159 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" event={"ID":"5c49724a-9d0f-42ec-86c8-cf8f7229f38a","Type":"ContainerStarted","Data":"3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3"} Nov 04 01:28:42.149419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:42.149397 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:28:42.151632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:42.151610 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" event={"ID":"c9f5857b-b55b-4002-b4a3-4f61950bd6ba","Type":"ContainerStarted","Data":"1e45be6dce63a12350b29bf107b3657cb75ac0992129dc2ecd622a562c0c9419"} Nov 04 01:28:42.151771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:42.151758 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:28:43.452511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:43.452467 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="235a54ac-8745-4f3b-bf01-d14af58c95c3" path="/var/lib/kubelet/pods/235a54ac-8745-4f3b-bf01-d14af58c95c3/volumes" Nov 04 01:28:43.561356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:43.501501 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="700b6217-0c45-4b58-9ddc-4b0ba67c5505" path="/var/lib/kubelet/pods/700b6217-0c45-4b58-9ddc-4b0ba67c5505/volumes" Nov 04 01:28:43.758263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:43.758223 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-69d5854bb-87mwv" Nov 04 01:28:44.216278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:44.053027 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7"] Nov 04 01:28:44.216278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:44.064872 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c00ce92-qwgf7"] Nov 04 01:28:44.216278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:44.168564 5051 scope.go:122] "RemoveContainer" containerID="60e0e9402c7566f11a446dabdad8fc3c709ba34f8e265e6e1e7ab3c9122401c0" Nov 04 01:28:44.891040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:44.891015 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k"] Nov 04 01:28:44.897557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:44.897523 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6"] Nov 04 01:28:44.903788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:44.903750 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh"] Nov 04 01:28:44.908839 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:44.908815 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc"] Nov 04 01:28:44.913641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:44.913619 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh"] Nov 04 01:28:45.009206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.009181 5051 scope.go:122] "RemoveContainer" containerID="5d1c9e59017c71fd0b0e925959a66689f8d9e4d984a1086da751c22b428e8701" Nov 04 01:28:45.205602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.205512 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0530ca-01be-4160-8f8f-4c7c778fe169" path="/var/lib/kubelet/pods/ad0530ca-01be-4160-8f8f-4c7c778fe169/volumes" Nov 04 01:28:45.222394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.222366 5051 scope.go:122] "RemoveContainer" containerID="f96873fe4379b7b2dc849f1d072d760e8d716672be902c88ae9c94a6bf15bd94" Nov 04 01:28:45.230498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.230450 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.230645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.230624 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.231618 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.230787 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.232196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.232150 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.232617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.232291 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.232617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.232375 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.278332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.275637 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.278332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.275839 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.278332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.275951 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.392417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.388107 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.392417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.388574 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.392417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.388875 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:45.404330 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.404173 5051 scope.go:122] "RemoveContainer" containerID="5a1180f3c4514c36dac4d3e45353fc33f9acedab38fa166657a4ee545ed2ec6b" Nov 04 01:28:45.437477 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.437452 5051 scope.go:122] "RemoveContainer" containerID="afc7798f538e610ef92f877404fcee37f457f680af4906078f8eea012971f876" Nov 04 01:28:45.494690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.494593 5051 scope.go:122] "RemoveContainer" containerID="c0e511ab0b1af42761b0b10f416c3f69cc50587b0d6c3dc5a4d4e0cbe78152eb" Nov 04 01:28:45.528102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:45.527983 5051 scope.go:122] "RemoveContainer" containerID="ed92b74ca2197ccce3a22e9c7673a124da098563d3a3ab27cb36bc477bd17a43" Nov 04 01:28:46.401337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.401301 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" event={"ID":"3631a39e-5bb5-4de8-9fa2-2830f3f00a94","Type":"ContainerStarted","Data":"2f0ffce8ce0c12d65173062cd7d74c4a1fccde75ebb2da2355d7669843e7ab40"} Nov 04 01:28:46.401867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.401368 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" event={"ID":"3631a39e-5bb5-4de8-9fa2-2830f3f00a94","Type":"ContainerStarted","Data":"06bd316ae8b402b1520702f5151e9103914f67594c2732a6e3584fc6f69ac426"} Nov 04 01:28:46.403378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.403344 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" event={"ID":"77c4ea51-856e-4992-bc91-d291fa962765","Type":"ContainerStarted","Data":"513605656f043901892bea4bedf8440265de2d36a3be978237ea975abc5d2c8f"} Nov 04 01:28:46.403525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.403400 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" event={"ID":"77c4ea51-856e-4992-bc91-d291fa962765","Type":"ContainerStarted","Data":"b62d29a0564cd9043e7bc5637b817d24b179399b29e1583d59f976ea515f7ac7"} Nov 04 01:28:46.403525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.403432 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:46.405352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.405316 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" event={"ID":"9540afba-e788-48cf-9e8e-b8a685b2f8b6","Type":"ContainerStarted","Data":"f0cf73aa2fc6bd8275026fa45aa75a69a9142753a0e99a91684369a6d820b50c"} Nov 04 01:28:46.405483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.405364 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" event={"ID":"9540afba-e788-48cf-9e8e-b8a685b2f8b6","Type":"ContainerStarted","Data":"f7ec83c532aeff3d6fc1039b9967fef85c92cb842391e8ff6fb1fce588cca0f0"} Nov 04 01:28:46.407099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.407076 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" event={"ID":"e1c36e46-07ec-47df-a65c-d35233518e01","Type":"ContainerStarted","Data":"45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a"} Nov 04 01:28:46.407240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.407165 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" event={"ID":"e1c36e46-07ec-47df-a65c-d35233518e01","Type":"ContainerStarted","Data":"2f84f2620289143ed702edc53a3d484f6c7ebd240b3d7be9c9a7bb904eeda04c"} Nov 04 01:28:46.408917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.408894 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" event={"ID":"fcb153fc-0c8d-4625-8b59-36344e1f3766","Type":"ContainerStarted","Data":"7b37c817fe94bfa4ff89de1fd59c18438775864fd61fdf249bfa79c14cbcb3df"} Nov 04 01:28:46.409010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.408931 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" event={"ID":"fcb153fc-0c8d-4625-8b59-36344e1f3766","Type":"ContainerStarted","Data":"64a2d3aac62a861656d45303fdd39677106bcec61f4101c8e2be238dd76acf7f"} Nov 04 01:28:46.428875 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.428795 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podStartSLOduration=24.428779101 podStartE2EDuration="24.428779101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:46.42327357 +0000 UTC m=+1865.395022445" watchObservedRunningTime="2026-11-04 01:28:46.428779101 +0000 UTC m=+1865.400527974" Nov 04 01:28:46.449236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.449191 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" podStartSLOduration=46.449179218 podStartE2EDuration="46.449179218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:46.445041762 +0000 UTC m=+1865.416790636" watchObservedRunningTime="2026-11-04 01:28:46.449179218 +0000 UTC m=+1865.420928089" Nov 04 01:28:46.464465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.464416 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" podStartSLOduration=7.464404667 podStartE2EDuration="7.464404667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:46.463552705 +0000 UTC m=+1865.435301576" watchObservedRunningTime="2026-11-04 01:28:46.464404667 +0000 UTC m=+1865.436153538" Nov 04 01:28:46.490575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.490528 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" podStartSLOduration=31.490515806 podStartE2EDuration="31.490515806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:46.483555614 +0000 UTC m=+1865.455304508" watchObservedRunningTime="2026-11-04 01:28:46.490515806 +0000 UTC m=+1865.462264677" Nov 04 01:28:46.505885 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.505843 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podStartSLOduration=30.505830626 podStartE2EDuration="30.505830626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:46.50342813 +0000 UTC m=+1865.475177001" watchObservedRunningTime="2026-11-04 01:28:46.505830626 +0000 UTC m=+1865.477579556" Nov 04 01:28:46.580229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.579925 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:46.581377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.581295 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:28:46.581377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.581364 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:28:46.742879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.742383 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:46.742879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.742500 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 04 01:28:46.742879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.742548 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" podUID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 04 01:28:46.928657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.920950 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:46.928657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.922395 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:28:46.928657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:46.922449 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:28:47.087192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:47.080620 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:47.097150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:47.091240 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 04 01:28:47.097150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:47.091308 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" podUID="fcb153fc-0c8d-4625-8b59-36344e1f3766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 04 01:28:47.584601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:47.584574 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:47.740302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:47.740265 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 04 01:28:47.740487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:47.740328 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" podUID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 04 01:28:47.925522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:47.925445 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:48.084499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:48.084459 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:48.429555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:48.429513 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:48.429757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:48.429616 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:48.429757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:48.429710 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:48.430648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:48.430625 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:28:48.430837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:48.430678 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:28:48.430837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:48.430708 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:28:48.745550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:48.745522 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:49.384588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:49.382257 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:28:49.384588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:49.382332 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:28:49.384588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:49.382453 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:28:49.442316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:49.442254 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:49.443512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:49.443478 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:28:49.649095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:49.649040 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": read tcp 10.132.0.2:52968->10.132.0.155:15021: read: connection reset by peer" start-of-body= Nov 04 01:28:49.649329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:49.649099 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": read tcp 10.132.0.2:52968->10.132.0.155:15021: read: connection reset by peer" Nov 04 01:28:50.445447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.445411 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8"] Nov 04 01:28:50.450454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.450420 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:50.454013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.453984 5051 generic.go:356] "Generic (PLEG): container finished" podID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerID="5170c75c442c95a9d06c312d180443dcf6dec2ec9e051c9b36e05411e4c910fc" exitCode=0 Nov 04 01:28:50.454157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.454059 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" event={"ID":"747c4b7d-4c96-42ab-b459-8eb5aaf75e09","Type":"ContainerDied","Data":"5170c75c442c95a9d06c312d180443dcf6dec2ec9e051c9b36e05411e4c910fc"} Nov 04 01:28:50.455579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.454488 5051 scope.go:122] "RemoveContainer" containerID="5170c75c442c95a9d06c312d180443dcf6dec2ec9e051c9b36e05411e4c910fc" Nov 04 01:28:50.456233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.456206 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:50.456293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.456272 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:50.456335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.456315 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:28:50.481313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.481283 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df6da4c0\"" Nov 04 01:28:50.481418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.481388 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df6da4c0-dockercfg-v29w4\"" Nov 04 01:28:50.506645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.506597 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8"] Nov 04 01:28:50.555250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.548391 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8\" (UID: \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\") " pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:50.555250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.549031 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrfxs\" (UniqueName: \"kubernetes.io/projected/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-kube-api-access-nrfxs\") pod \"payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8\" (UID: \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\") " pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:50.689862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.689809 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8\" (UID: \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\") " pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:50.718432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.703716 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrfxs\" (UniqueName: \"kubernetes.io/projected/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-kube-api-access-nrfxs\") pod \"payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8\" (UID: \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\") " pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:50.732155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.731579 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8\" (UID: \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\") " pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:51.002655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:50.992347 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrfxs\" (UniqueName: \"kubernetes.io/projected/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-kube-api-access-nrfxs\") pod \"payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8\" (UID: \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\") " pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:51.066954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.066915 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:51.126163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.124598 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7"] Nov 04 01:28:51.157456 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.157426 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e2fb6d-lzzd7"] Nov 04 01:28:51.220139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.216199 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f27d705-1a3c-44d3-99b2-126371818776" path="/var/lib/kubelet/pods/2f27d705-1a3c-44d3-99b2-126371818776/volumes" Nov 04 01:28:51.376108 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:28:51.376063 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbefb61b_8223_4eac_a2c0_bb9b4d2913fc.slice/crio-d17d91e6e09aa31dc0d25d60ba158e3618d81a0797a596f8da080b46e50986bf WatchSource:0}: Error finding container d17d91e6e09aa31dc0d25d60ba158e3618d81a0797a596f8da080b46e50986bf: Status 404 returned error can't find the container with id d17d91e6e09aa31dc0d25d60ba158e3618d81a0797a596f8da080b46e50986bf Nov 04 01:28:51.378209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.378177 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8"] Nov 04 01:28:51.385850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.385821 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx"] Nov 04 01:28:51.393308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.393283 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p"] Nov 04 01:28:51.393637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.393604 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:51.404620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.404582 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:51.408679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.407822 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-75f7ae\"" Nov 04 01:28:51.408679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.407935 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-75f7ae-dockercfg-gqnqs\"" Nov 04 01:28:51.409562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.409539 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p"] Nov 04 01:28:51.418280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.418253 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx"] Nov 04 01:28:51.466829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.466795 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" event={"ID":"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc","Type":"ContainerStarted","Data":"d17d91e6e09aa31dc0d25d60ba158e3618d81a0797a596f8da080b46e50986bf"} Nov 04 01:28:51.469659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.469618 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" event={"ID":"747c4b7d-4c96-42ab-b459-8eb5aaf75e09","Type":"ContainerStarted","Data":"c29baaba157ffc827f199126f19f3910f423cb182a6af935e8ef4c437cbc5e73"} Nov 04 01:28:51.508286 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.508261 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7hd\" (UniqueName: \"kubernetes.io/projected/5c8c1b3e-1289-4c31-89d0-e43168ec9619-kube-api-access-7q7hd\") pod \"payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx\" (UID: \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:51.508408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.508321 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8c1b3e-1289-4c31-89d0-e43168ec9619-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx\" (UID: \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:51.508591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.508562 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b86f87c-1d12-4718-b03f-a32375cf87a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p\" (UID: \"5b86f87c-1d12-4718-b03f-a32375cf87a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:51.508649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.508622 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjvb\" (UniqueName: \"kubernetes.io/projected/5b86f87c-1d12-4718-b03f-a32375cf87a5-kube-api-access-4fjvb\") pod \"payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p\" (UID: \"5b86f87c-1d12-4718-b03f-a32375cf87a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:51.644196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.636611 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7hd\" (UniqueName: \"kubernetes.io/projected/5c8c1b3e-1289-4c31-89d0-e43168ec9619-kube-api-access-7q7hd\") pod \"payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx\" (UID: \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:51.644196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.636719 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8c1b3e-1289-4c31-89d0-e43168ec9619-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx\" (UID: \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:51.644196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.636877 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b86f87c-1d12-4718-b03f-a32375cf87a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p\" (UID: \"5b86f87c-1d12-4718-b03f-a32375cf87a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:51.644196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.636920 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjvb\" (UniqueName: \"kubernetes.io/projected/5b86f87c-1d12-4718-b03f-a32375cf87a5-kube-api-access-4fjvb\") pod \"payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p\" (UID: \"5b86f87c-1d12-4718-b03f-a32375cf87a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:51.645711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.645394 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b86f87c-1d12-4718-b03f-a32375cf87a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p\" (UID: \"5b86f87c-1d12-4718-b03f-a32375cf87a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:51.653161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.646752 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8c1b3e-1289-4c31-89d0-e43168ec9619-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx\" (UID: \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:51.715145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.711495 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjvb\" (UniqueName: \"kubernetes.io/projected/5b86f87c-1d12-4718-b03f-a32375cf87a5-kube-api-access-4fjvb\") pod \"payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p\" (UID: \"5b86f87c-1d12-4718-b03f-a32375cf87a5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:51.743144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.728302 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7hd\" (UniqueName: \"kubernetes.io/projected/5c8c1b3e-1289-4c31-89d0-e43168ec9619-kube-api-access-7q7hd\") pod \"payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx\" (UID: \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:51.785767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:51.782441 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:52.025473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.025435 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:52.100953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.100889 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn"] Nov 04 01:28:52.116076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.116035 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr"] Nov 04 01:28:52.122322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.122296 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e77e7540-28vmr"] Nov 04 01:28:52.134980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.134944 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e5e5c5-j5hkn"] Nov 04 01:28:52.235279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.220662 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:28:52.235279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.220807 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 04 01:28:52.235279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.220863 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 04 01:28:52.632916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.632712 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx"] Nov 04 01:28:52.670729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.664251 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" event={"ID":"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc","Type":"ContainerStarted","Data":"90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a"} Nov 04 01:28:52.670729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.664384 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:28:52.675770 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:28:52.675733 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8c1b3e_1289_4c31_89d0_e43168ec9619.slice/crio-9bb08f9ac0ed10ac2d747ada126131f541c9d0ead70c4d542d5d1b620b29e931 WatchSource:0}: Error finding container 9bb08f9ac0ed10ac2d747ada126131f541c9d0ead70c4d542d5d1b620b29e931: Status 404 returned error can't find the container with id 9bb08f9ac0ed10ac2d747ada126131f541c9d0ead70c4d542d5d1b620b29e931 Nov 04 01:28:52.705134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.705070 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" podStartSLOduration=2.705057242 podStartE2EDuration="2.705057242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:52.699735564 +0000 UTC m=+1871.671484440" watchObservedRunningTime="2026-11-04 01:28:52.705057242 +0000 UTC m=+1871.676806114" Nov 04 01:28:52.793459 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:28:52.793425 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b86f87c_1d12_4718_b03f_a32375cf87a5.slice/crio-e90da0a459651a78c9973c784ffe245ae5707f99a7e419a2495efdb66c3cbbc2 WatchSource:0}: Error finding container e90da0a459651a78c9973c784ffe245ae5707f99a7e419a2495efdb66c3cbbc2: Status 404 returned error can't find the container with id e90da0a459651a78c9973c784ffe245ae5707f99a7e419a2495efdb66c3cbbc2 Nov 04 01:28:52.794455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:52.794428 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p"] Nov 04 01:28:53.276227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.276165 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f139661f-18a5-432b-8b79-cfb1302f37d6" path="/var/lib/kubelet/pods/f139661f-18a5-432b-8b79-cfb1302f37d6/volumes" Nov 04 01:28:53.281200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.277115 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe36d7d9-13ec-4ec7-b015-67f700ecb66f" path="/var/lib/kubelet/pods/fe36d7d9-13ec-4ec7-b015-67f700ecb66f/volumes" Nov 04 01:28:53.281200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.278409 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:28:53.281200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.278620 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-tmmwg" Nov 04 01:28:53.281200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.278737 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:28:53.761150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.760612 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" event={"ID":"5b86f87c-1d12-4718-b03f-a32375cf87a5","Type":"ContainerStarted","Data":"bcde942b5da34a582b28c4c96c6cca53d4e9cb3a0063bfe1b69fafee331a6fb2"} Nov 04 01:28:53.761150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.760743 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" event={"ID":"5b86f87c-1d12-4718-b03f-a32375cf87a5","Type":"ContainerStarted","Data":"e90da0a459651a78c9973c784ffe245ae5707f99a7e419a2495efdb66c3cbbc2"} Nov 04 01:28:53.765143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.762151 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:28:53.793902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.791609 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" event={"ID":"5c8c1b3e-1289-4c31-89d0-e43168ec9619","Type":"ContainerStarted","Data":"741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39"} Nov 04 01:28:53.793902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.791674 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:28:53.793902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.791701 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" event={"ID":"5c8c1b3e-1289-4c31-89d0-e43168ec9619","Type":"ContainerStarted","Data":"9bb08f9ac0ed10ac2d747ada126131f541c9d0ead70c4d542d5d1b620b29e931"} Nov 04 01:28:53.793902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.791723 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:28:53.813144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.803479 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:28:53.815065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.814992 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" podStartSLOduration=2.814808445 podStartE2EDuration="2.814808445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:53.798678087 +0000 UTC m=+1872.770426958" watchObservedRunningTime="2026-11-04 01:28:53.814808445 +0000 UTC m=+1872.786557319" Nov 04 01:28:53.880215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:53.880158 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" podStartSLOduration=2.880139216 podStartE2EDuration="2.880139216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:28:53.871721431 +0000 UTC m=+1872.843470306" watchObservedRunningTime="2026-11-04 01:28:53.880139216 +0000 UTC m=+1872.851888087" Nov 04 01:28:57.422899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:57.422863 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:28:58.028020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:58.027987 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g"] Nov 04 01:28:58.033395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:58.033364 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa640c3a-w6q6g"] Nov 04 01:28:59.191608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:28:59.191569 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afc777e-5c73-4cd5-bc31-08ca02fc809a" path="/var/lib/kubelet/pods/9afc777e-5c73-4cd5-bc31-08ca02fc809a/volumes" Nov 04 01:29:01.129531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:01.129496 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt"] Nov 04 01:29:01.139750 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:01.139718 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19ee9690-tbbdt"] Nov 04 01:29:01.387925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:01.387840 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b492feea-9e67-4e2a-aed9-8e9c3386e82a" path="/var/lib/kubelet/pods/b492feea-9e67-4e2a-aed9-8e9c3386e82a/volumes" Nov 04 01:29:02.123172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:02.121193 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts"] Nov 04 01:29:02.143234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:02.129314 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147c2eaf-rdtts"] Nov 04 01:29:03.251145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:03.243869 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7fe3ad-dbe8-42d9-bb41-722519b05069" path="/var/lib/kubelet/pods/9a7fe3ad-dbe8-42d9-bb41-722519b05069/volumes" Nov 04 01:29:03.797297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:03.797262 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:29:04.557425 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:29:04.557386 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:29:04.881196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:04.880591 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:29:05.884144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:05.882900 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:29:06.056576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:06.056540 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d"] Nov 04 01:29:06.062069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:06.062045 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a35b987-jmw8d"] Nov 04 01:29:07.190347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:07.190311 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b994d45-a1e2-4058-a193-49b30619dfc3" path="/var/lib/kubelet/pods/2b994d45-a1e2-4058-a193-49b30619dfc3/volumes" Nov 04 01:29:19.032272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:19.032237 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr"] Nov 04 01:29:19.037405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:19.037378 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e42b7f-nhjdr"] Nov 04 01:29:19.268308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:19.268269 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc53491e-7a47-49b1-86b3-d47440df1135" path="/var/lib/kubelet/pods/bc53491e-7a47-49b1-86b3-d47440df1135/volumes" Nov 04 01:29:21.584147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.581685 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:21.847007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.846922 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb"] Nov 04 01:29:21.914435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.914379 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:21.923468 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.923426 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb"] Nov 04 01:29:21.926628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.926601 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:29:21.927572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.927549 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:29:21.949153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.949095 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8zv\" (UniqueName: \"kubernetes.io/projected/e5c74efd-d906-43ac-9bea-293d9cde3005-kube-api-access-9s8zv\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:21.949725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.949699 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:21.950317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:21.950294 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:22.004090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.004060 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:29:22.004550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.004524 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:29:22.051693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.051652 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:22.051892 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.051717 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8zv\" (UniqueName: \"kubernetes.io/projected/e5c74efd-d906-43ac-9bea-293d9cde3005-kube-api-access-9s8zv\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:22.051892 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.051819 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:22.052635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.052604 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:22.056273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.056246 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:22.070931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.070888 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8zv\" (UniqueName: \"kubernetes.io/projected/e5c74efd-d906-43ac-9bea-293d9cde3005-kube-api-access-9s8zv\") pod \"maas-api-revoke-sub-a2747496-dw9cb\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:22.277150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:22.273955 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:23.089841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:23.089777 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j"] Nov 04 01:29:23.111347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:23.111309 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d8aa4aa-wh85j"] Nov 04 01:29:23.210691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:23.210649 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36359561-fa5d-4c84-8e13-bbd1ab48ab40" path="/var/lib/kubelet/pods/36359561-fa5d-4c84-8e13-bbd1ab48ab40/volumes" Nov 04 01:29:23.220735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:23.220708 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb"] Nov 04 01:29:24.117611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:24.117558 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" event={"ID":"e5c74efd-d906-43ac-9bea-293d9cde3005","Type":"ContainerStarted","Data":"ef97dd611ffabaf74464f56926014ae02920ef7ea59d8ab624ecaabca51a8142"} Nov 04 01:29:24.910427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:24.910380 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:25.047674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.047248 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:25.057252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.057212 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:25.320469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.320426 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:25.341262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.341213 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:29:25.377475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.376831 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr"] Nov 04 01:29:25.377742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.377717 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" event={"ID":"e5c74efd-d906-43ac-9bea-293d9cde3005","Type":"ContainerStarted","Data":"564494348f5efde3da2fcb9721d08b2e46ee51b95cda91d8ba67aa5310f033c4"} Nov 04 01:29:25.490332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.490290 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f6ff71-z4nqr"] Nov 04 01:29:25.551171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.533807 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" podStartSLOduration=4.5337853599999995 podStartE2EDuration="4.53378536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:25.382043777 +0000 UTC m=+1904.353792649" watchObservedRunningTime="2026-11-04 01:29:25.53378536 +0000 UTC m=+1904.505534231" Nov 04 01:29:25.797203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.796665 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:25.840156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:25.837237 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:29:26.085209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.070286 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:29:26.085209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.070355 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:26.085209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.070772 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:29:26.085209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.070811 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:26.378201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.377651 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz"] Nov 04 01:29:26.742208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.725723 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:26.807144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.804521 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:26.807144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.805104 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:29:26.927417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:26.919490 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:27.034518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.034402 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:29:27.035524 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.035491 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:27.113573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.113534 5051 generic.go:356] "Generic (PLEG): container finished" podID="e5c74efd-d906-43ac-9bea-293d9cde3005" containerID="564494348f5efde3da2fcb9721d08b2e46ee51b95cda91d8ba67aa5310f033c4" exitCode=0 Nov 04 01:29:27.238952 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.166576 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" event={"ID":"e5c74efd-d906-43ac-9bea-293d9cde3005","Type":"ContainerDied","Data":"564494348f5efde3da2fcb9721d08b2e46ee51b95cda91d8ba67aa5310f033c4"} Nov 04 01:29:27.337436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.228412 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": context deadline exceeded" Nov 04 01:29:27.337859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.337821 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:27.338045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.283458 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:29:27.339332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.238684 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:27.896148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.894319 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:29:27.984145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:27.976220 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:29:28.040146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.037295 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:29:28.040146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.038528 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:29:28.040146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.039761 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:28.436177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.411219 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="557acf14-07aa-4b15-9ce8-6e48c960755e" path="/var/lib/kubelet/pods/557acf14-07aa-4b15-9ce8-6e48c960755e/volumes" Nov 04 01:29:28.663147 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:29:28.636423 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.348s" Nov 04 01:29:28.825167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.783960 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf"] Nov 04 01:29:28.869167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.467347 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:28.869167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.619664 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:28.869167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.462407 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:29:28.915145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:28.912203 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:29:29.065147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.064921 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:29:29.136200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.125879 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:29:29.136200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.126336 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:29:29.136200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.126775 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:29:29.447443 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.447217 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:29.457458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.457429 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf"] Nov 04 01:29:29.457722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.457705 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:29:29.457869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.457855 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88"] Nov 04 01:29:29.497410 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.457579 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:29.509154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.474273 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:29.546151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.540473 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf\" (UID: \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:29.589371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.589338 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88"] Nov 04 01:29:29.709197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.644357 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:29.967208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.894750 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qknb\" (UniqueName: \"kubernetes.io/projected/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-kube-api-access-4qknb\") pod \"payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf\" (UID: \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.919278 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.936443 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-5f572a-dockercfg-7kz7c\"" Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.938988 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-5f572a\"" Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.982665 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.982708 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm"] Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.982750 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.982785 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.982823 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:29:29.988023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:29.982855 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:29:30.006250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.006214 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:30.059715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.038699 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:30.059715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.039069 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:30.059715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.039356 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:29:30.110145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.100790 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:29:30.143975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.123408 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531cfc6a-8853-4788-8d9a-c05d9a233639-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88\" (UID: \"531cfc6a-8853-4788-8d9a-c05d9a233639\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:30.143975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.123559 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5z2\" (UniqueName: \"kubernetes.io/projected/531cfc6a-8853-4788-8d9a-c05d9a233639-kube-api-access-jw5z2\") pod \"payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88\" (UID: \"531cfc6a-8853-4788-8d9a-c05d9a233639\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:30.143975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.123809 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf\" (UID: \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:30.143975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.124145 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qknb\" (UniqueName: \"kubernetes.io/projected/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-kube-api-access-4qknb\") pod \"payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf\" (UID: \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:30.154977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.154942 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf\" (UID: \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:30.185155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.183771 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:29:30.185155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.184157 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Nov 04 01:29:30.185155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.184440 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:29:30.193151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.186420 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65d4ea46-4fvpm"] Nov 04 01:29:30.235760 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.235706 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531cfc6a-8853-4788-8d9a-c05d9a233639-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88\" (UID: \"531cfc6a-8853-4788-8d9a-c05d9a233639\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:30.316156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.277401 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5z2\" (UniqueName: \"kubernetes.io/projected/531cfc6a-8853-4788-8d9a-c05d9a233639-kube-api-access-jw5z2\") pod \"payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88\" (UID: \"531cfc6a-8853-4788-8d9a-c05d9a233639\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:30.399483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.276231 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531cfc6a-8853-4788-8d9a-c05d9a233639-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88\" (UID: \"531cfc6a-8853-4788-8d9a-c05d9a233639\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:30.444171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.441927 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qknb\" (UniqueName: \"kubernetes.io/projected/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-kube-api-access-4qknb\") pod \"payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf\" (UID: \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:30.535678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.478230 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5z2\" (UniqueName: \"kubernetes.io/projected/531cfc6a-8853-4788-8d9a-c05d9a233639-kube-api-access-jw5z2\") pod \"payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88\" (UID: \"531cfc6a-8853-4788-8d9a-c05d9a233639\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:30.709146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:30.629839 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:31.097314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.047023 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:31.097314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.047441 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:31.097314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.061366 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:31.105093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.105050 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:29:31.105961 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.105929 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:29:31.170297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.141932 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd"] Nov 04 01:29:31.170297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.142015 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:29:31.535509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.535465 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:29:31.556607 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:29:31.556571 5051 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-40e43d6e-452vd\" is forbidden: User \"system:node:ip-10-0-137-48.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-137-48.us-east-2.compute.internal' and this object" podUID="02116ac0-a24a-4c8f-a87c-e206df67e1ee" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd" Nov 04 01:29:31.676422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.676361 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:32.009474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.944676 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88bf7c0d-6fcd-46a0-80f4-cd39c247e483" path="/var/lib/kubelet/pods/88bf7c0d-6fcd-46a0-80f4-cd39c247e483/volumes" Nov 04 01:29:32.009474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.945381 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:29:32.009474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.945421 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40e43d6e-452vd"] Nov 04 01:29:32.009474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.945452 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn"] Nov 04 01:29:32.009474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.945730 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:29:32.009474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.945784 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:29:32.009474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.952855 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:29:32.009474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:31.956799 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" containerID="cri-o://7df72b0c28079c8516b47ed377cb4382e2906345e38c8562b46e1aef22e9afdc" gracePeriod=30 Nov 04 01:29:32.016370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.010003 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:29:32.016370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.011106 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:29:32.144520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.144435 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd"] Nov 04 01:29:32.214262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.212481 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:32.403461 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.403345 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5c4e262-rjdsd"] Nov 04 01:29:32.543196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.543149 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:32.606111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.606071 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-xks-ca\") pod \"e5c74efd-d906-43ac-9bea-293d9cde3005\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " Nov 04 01:29:32.606410 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.606377 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-openshift-ca\") pod \"e5c74efd-d906-43ac-9bea-293d9cde3005\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " Nov 04 01:29:32.606624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.606604 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c74efd-d906-43ac-9bea-293d9cde3005-kube-api-access-9s8zv\" (UniqueName: \"kubernetes.io/projected/e5c74efd-d906-43ac-9bea-293d9cde3005-kube-api-access-9s8zv\") pod \"e5c74efd-d906-43ac-9bea-293d9cde3005\" (UID: \"e5c74efd-d906-43ac-9bea-293d9cde3005\") " Nov 04 01:29:32.607635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.607610 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:29:32.610336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.609897 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-openshift-ca" pod "e5c74efd-d906-43ac-9bea-293d9cde3005" (UID: "e5c74efd-d906-43ac-9bea-293d9cde3005"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:32.729259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.729206 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:32.804320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.795969 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-xks-ca" pod "e5c74efd-d906-43ac-9bea-293d9cde3005" (UID: "e5c74efd-d906-43ac-9bea-293d9cde3005"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:32.913251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:32.913217 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5c74efd-d906-43ac-9bea-293d9cde3005-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:33.270140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:33.266360 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" Nov 04 01:29:33.351664 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:33.351625 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c74efd-d906-43ac-9bea-293d9cde3005-kube-api-access-9s8zv" pod "e5c74efd-d906-43ac-9bea-293d9cde3005" (UID: "e5c74efd-d906-43ac-9bea-293d9cde3005"). InnerVolumeSpecName "kube-api-access-9s8zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:33.442763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:33.442668 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s8zv\" (UniqueName: \"kubernetes.io/projected/e5c74efd-d906-43ac-9bea-293d9cde3005-kube-api-access-9s8zv\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:33.738500 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:29:33.737249 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531cfc6a_8853_4788_8d9a_c05d9a233639.slice/crio-ba974558d575df6e383ee1c3550222813ed621903cf2a72f3925701e6a1ba107 WatchSource:0}: Error finding container ba974558d575df6e383ee1c3550222813ed621903cf2a72f3925701e6a1ba107: Status 404 returned error can't find the container with id ba974558d575df6e383ee1c3550222813ed621903cf2a72f3925701e6a1ba107 Nov 04 01:29:33.774143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:33.749161 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02116ac0-a24a-4c8f-a87c-e206df67e1ee" path="/var/lib/kubelet/pods/02116ac0-a24a-4c8f-a87c-e206df67e1ee/volumes" Nov 04 01:29:34.129988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:34.129574 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0a58ec-6d24-4037-973b-4450e76ce371" path="/var/lib/kubelet/pods/1c0a58ec-6d24-4037-973b-4450e76ce371/volumes" Nov 04 01:29:34.253382 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:29:34.245578 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.06s" Nov 04 01:29:34.253382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:34.245632 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb" event={"ID":"e5c74efd-d906-43ac-9bea-293d9cde3005","Type":"ContainerDied","Data":"ef97dd611ffabaf74464f56926014ae02920ef7ea59d8ab624ecaabca51a8142"} Nov 04 01:29:34.253382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:34.245658 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef97dd611ffabaf74464f56926014ae02920ef7ea59d8ab624ecaabca51a8142" Nov 04 01:29:34.253382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:34.245712 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88"] Nov 04 01:29:34.347532 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:29:34.347497 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:29:34.424858 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:34.424826 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf"] Nov 04 01:29:34.451149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:34.449071 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" event={"ID":"531cfc6a-8853-4788-8d9a-c05d9a233639","Type":"ContainerStarted","Data":"ba974558d575df6e383ee1c3550222813ed621903cf2a72f3925701e6a1ba107"} Nov 04 01:29:35.486999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:35.482662 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" event={"ID":"bac07c86-7302-4f3e-a71b-ac3adcb85fb0","Type":"ContainerStarted","Data":"784e99b3b418341c3278eb5df09dabe8bfd71ff472db00e4db31cbabc6f142f8"} Nov 04 01:29:35.486999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:35.482762 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" event={"ID":"bac07c86-7302-4f3e-a71b-ac3adcb85fb0","Type":"ContainerStarted","Data":"bc735fb73702fbdef1b042193e72f4445da1a167786f366e361818d1789690ff"} Nov 04 01:29:35.486999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:35.484079 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:35.494070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:35.493039 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" event={"ID":"531cfc6a-8853-4788-8d9a-c05d9a233639","Type":"ContainerStarted","Data":"8362a71bd2c2f7230cc5f75f89ed3bf4461c0aeaea9ab171f65892d85f6ce2d1"} Nov 04 01:29:35.494070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:35.494013 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:35.546600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:35.542369 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" podStartSLOduration=9.542343821 podStartE2EDuration="9.542343821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:35.510758544 +0000 UTC m=+1914.482507417" watchObservedRunningTime="2026-11-04 01:29:35.542343821 +0000 UTC m=+1914.514092694" Nov 04 01:29:35.606255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:35.602926 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" podStartSLOduration=9.602903091 podStartE2EDuration="9.602903091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:35.564454749 +0000 UTC m=+1914.536203673" watchObservedRunningTime="2026-11-04 01:29:35.602903091 +0000 UTC m=+1914.574651975" Nov 04 01:29:37.609979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.602370 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" event={"ID":"5541dbc1-541d-4e63-ab47-5dd90fe99545","Type":"ContainerDied","Data":"7df72b0c28079c8516b47ed377cb4382e2906345e38c8562b46e1aef22e9afdc"} Nov 04 01:29:37.609979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.602459 5051 scope.go:122] "RemoveContainer" containerID="8f90df9f1bf31845bf22fce2b65a3c43361e28757406ae4733e2d6fc7be8e7d4" Nov 04 01:29:37.609979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.602616 5051 generic.go:356] "Generic (PLEG): container finished" podID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerID="7df72b0c28079c8516b47ed377cb4382e2906345e38c8562b46e1aef22e9afdc" exitCode=0 Nov 04 01:29:37.609979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.602760 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" event={"ID":"5541dbc1-541d-4e63-ab47-5dd90fe99545","Type":"ContainerDied","Data":"1cd3de94ea8674a501c44f4446788b1507e491934e548bea926b5f5420f67fc4"} Nov 04 01:29:37.609979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.602800 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cd3de94ea8674a501c44f4446788b1507e491934e548bea926b5f5420f67fc4" Nov 04 01:29:37.616165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.615854 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:29:37.805307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805218 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-podinfo\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.805479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805334 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-credential-socket\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.805479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805388 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5541dbc1-541d-4e63-ab47-5dd90fe99545-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5541dbc1-541d-4e63-ab47-5dd90fe99545-istiod-ca-cert\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.805578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805481 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-kube-api-access-d8fhh\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-kube-api-access-d8fhh\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.805578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805543 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-envoy\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.805578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805573 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-data\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.805708 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805639 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-certs\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.805810 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805784 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-socket\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.805866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.805847 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-token\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-token\") pod \"5541dbc1-541d-4e63-ab47-5dd90fe99545\" (UID: \"5541dbc1-541d-4e63-ab47-5dd90fe99545\") " Nov 04 01:29:37.807843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.807812 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-certs" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:37.809682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.809497 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-data" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:37.811073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.809809 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-credential-socket" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:37.811073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.810089 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5541dbc1-541d-4e63-ab47-5dd90fe99545-istiod-ca-cert" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:37.813072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.813041 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-socket" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:37.815764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.815719 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-envoy" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:37.818056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.818016 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-podinfo" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:29:37.819282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.819261 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-token" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:37.819507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.819484 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-kube-api-access-d8fhh" pod "5541dbc1-541d-4e63-ab47-5dd90fe99545" (UID: "5541dbc1-541d-4e63-ab47-5dd90fe99545"). InnerVolumeSpecName "kube-api-access-d8fhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914591 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8fhh\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-kube-api-access-d8fhh\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914629 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914645 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914659 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914675 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914691 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914705 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5541dbc1-541d-4e63-ab47-5dd90fe99545-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914719 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5541dbc1-541d-4e63-ab47-5dd90fe99545-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:37.914758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:37.914734 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5541dbc1-541d-4e63-ab47-5dd90fe99545-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:38.619179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:38.619135 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn" Nov 04 01:29:38.686481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:38.686313 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn"] Nov 04 01:29:38.698402 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:38.698364 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-694vn"] Nov 04 01:29:39.261049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:39.261001 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" path="/var/lib/kubelet/pods/5541dbc1-541d-4e63-ab47-5dd90fe99545/volumes" Nov 04 01:29:40.051157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:40.049847 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t"] Nov 04 01:29:40.055249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:40.055206 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g"] Nov 04 01:29:40.062220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:40.062115 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5868b6c3-bxt2g"] Nov 04 01:29:40.071214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:40.071187 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ec3e095-kn46t"] Nov 04 01:29:41.340741 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:41.340687 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f658636-2519-4a3d-88d5-e1a307648713" path="/var/lib/kubelet/pods/6f658636-2519-4a3d-88d5-e1a307648713/volumes" Nov 04 01:29:41.440934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:41.440861 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b192653-e694-480f-b460-d3e77cd65942" path="/var/lib/kubelet/pods/8b192653-e694-480f-b460-d3e77cd65942/volumes" Nov 04 01:29:42.431630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.431593 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6"] Nov 04 01:29:42.432486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.432465 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" Nov 04 01:29:42.432628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.432499 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" Nov 04 01:29:42.432628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.432520 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e5c74efd-d906-43ac-9bea-293d9cde3005" containerName="revoke-keys" Nov 04 01:29:42.432628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.432527 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c74efd-d906-43ac-9bea-293d9cde3005" containerName="revoke-keys" Nov 04 01:29:42.432894 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.432872 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" Nov 04 01:29:42.433082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.433059 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" Nov 04 01:29:42.433260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.433084 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c74efd-d906-43ac-9bea-293d9cde3005" containerName="revoke-keys" Nov 04 01:29:42.455453 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.455415 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.469095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.469068 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:29:42.469698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.469662 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6"] Nov 04 01:29:42.470048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.470029 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:29:42.470267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.470253 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:29:42.470441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.470428 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:29:42.495360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.495327 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.495475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.495387 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qdk\" (UniqueName: \"kubernetes.io/projected/1f7b38a6-9f62-4105-a347-a9b064d18ce1-kube-api-access-95qdk\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.495540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.495512 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.597156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.597082 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.597156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.597143 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qdk\" (UniqueName: \"kubernetes.io/projected/1f7b38a6-9f62-4105-a347-a9b064d18ce1-kube-api-access-95qdk\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.597414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.597225 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.609877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.598285 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.773020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.772991 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qdk\" (UniqueName: \"kubernetes.io/projected/1f7b38a6-9f62-4105-a347-a9b064d18ce1-kube-api-access-95qdk\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.777566 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.777534 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45d7a8cd-8llc6\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:42.821583 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:42.821553 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:43.786651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:43.786623 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6"] Nov 04 01:29:43.790977 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:29:43.790943 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7b38a6_9f62_4105_a347_a9b064d18ce1.slice/crio-8a1fe80606a96cdce656794ed9415685d2240fca53b4b8ab28b90e49f4bebaf0 WatchSource:0}: Error finding container 8a1fe80606a96cdce656794ed9415685d2240fca53b4b8ab28b90e49f4bebaf0: Status 404 returned error can't find the container with id 8a1fe80606a96cdce656794ed9415685d2240fca53b4b8ab28b90e49f4bebaf0 Nov 04 01:29:44.727499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:44.727454 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" event={"ID":"1f7b38a6-9f62-4105-a347-a9b064d18ce1","Type":"ContainerStarted","Data":"cf70477f5a627238340f036429f8106d2717fb9edcd68b4541c6fb24528d0c08"} Nov 04 01:29:44.727727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:44.727561 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" event={"ID":"1f7b38a6-9f62-4105-a347-a9b064d18ce1","Type":"ContainerStarted","Data":"8a1fe80606a96cdce656794ed9415685d2240fca53b4b8ab28b90e49f4bebaf0"} Nov 04 01:29:44.747731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:44.747667 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" podStartSLOduration=2.7476472899999997 podStartE2EDuration="2.74764729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:44.744329161 +0000 UTC m=+1923.716078033" watchObservedRunningTime="2026-11-04 01:29:44.74764729 +0000 UTC m=+1923.719396201" Nov 04 01:29:46.614488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.614456 5051 scope.go:122] "RemoveContainer" containerID="2d77a33b045253fad50f7b4053124851abd2bbb12bb7025e159aa6648f0d6f91" Nov 04 01:29:46.659206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.659176 5051 scope.go:122] "RemoveContainer" containerID="54fc9fcd0554d979139a38f7b620d2b86189dcb9c84734b0b2b9b4baa8b96b8e" Nov 04 01:29:46.767893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.767859 5051 generic.go:356] "Generic (PLEG): container finished" podID="1f7b38a6-9f62-4105-a347-a9b064d18ce1" containerID="cf70477f5a627238340f036429f8106d2717fb9edcd68b4541c6fb24528d0c08" exitCode=0 Nov 04 01:29:46.768062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.767938 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" event={"ID":"1f7b38a6-9f62-4105-a347-a9b064d18ce1","Type":"ContainerDied","Data":"cf70477f5a627238340f036429f8106d2717fb9edcd68b4541c6fb24528d0c08"} Nov 04 01:29:46.801601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.801573 5051 scope.go:122] "RemoveContainer" containerID="7df72b0c28079c8516b47ed377cb4382e2906345e38c8562b46e1aef22e9afdc" Nov 04 01:29:46.851190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.844832 5051 scope.go:122] "RemoveContainer" containerID="77f66924ba5a83b1d068ee39b6b07eafca13044d7906195e5e28cdc4f4e5117a" Nov 04 01:29:46.856092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.855091 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg"] Nov 04 01:29:46.953545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.953503 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" Nov 04 01:29:46.953545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:46.953537 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5541dbc1-541d-4e63-ab47-5dd90fe99545" containerName="istio-proxy" Nov 04 01:29:47.075326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.075294 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg"] Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.075373 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.101527 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgp8\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-kube-api-access-9kgp8\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.101585 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.101708 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.101757 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.101914 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.101941 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.102054 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.102096 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.109331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.102335 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.241690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.215835 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.241690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.215912 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.241690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.216169 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.241690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.216255 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgp8\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-kube-api-access-9kgp8\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.241690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.216333 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.241690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.216392 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.241690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.225268 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.252751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.252448 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.272498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.272463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.272775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.272730 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.272892 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.272782 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.312464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.312402 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.312653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.312459 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.314558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.312713 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.314558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.312747 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.314558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.313063 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.314558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.313108 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.314558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.313486 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgp8\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-kube-api-access-9kgp8\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.417815 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.417785 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:47.607626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.607597 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:29:47.608794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:47.608768 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:29:48.588094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:48.588049 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" containerID="cri-o://2b7d340067d428ddfae56ceb1c465eddb6333d3d278cd6b4d696ba409c9c5375" gracePeriod=38 Nov 04 01:29:49.056384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.030729 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": dial tcp 10.132.0.99:8000: connect: connection refused" Nov 04 01:29:49.362623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.339549 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg"] Nov 04 01:29:49.475297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.475266 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:29:49.475447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.475317 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:49.476241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.476215 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:29:49.476399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.476252 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:29:49.484424 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:29:49.484396 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cdea56_609e_4f8e_b99d_37d1e5b7f916.slice/crio-59bcd37f404a89e8fd3fe9ebf7edb8dfcbce4b412cb901b5f4cd73563ec28589 WatchSource:0}: Error finding container 59bcd37f404a89e8fd3fe9ebf7edb8dfcbce4b412cb901b5f4cd73563ec28589: Status 404 returned error can't find the container with id 59bcd37f404a89e8fd3fe9ebf7edb8dfcbce4b412cb901b5f4cd73563ec28589 Nov 04 01:29:49.487288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.487247 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:49.487407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.487325 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:49.487407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.487359 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:49.562063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.562035 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:49.645224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.645142 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-openshift-ca\") pod \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " Nov 04 01:29:49.645748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.645371 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f7b38a6-9f62-4105-a347-a9b064d18ce1-kube-api-access-95qdk\" (UniqueName: \"kubernetes.io/projected/1f7b38a6-9f62-4105-a347-a9b064d18ce1-kube-api-access-95qdk\") pod \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " Nov 04 01:29:49.645748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.645488 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-openshift-ca" pod "1f7b38a6-9f62-4105-a347-a9b064d18ce1" (UID: "1f7b38a6-9f62-4105-a347-a9b064d18ce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:29:49.645748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.645528 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-xks-ca\") pod \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\" (UID: \"1f7b38a6-9f62-4105-a347-a9b064d18ce1\") " Nov 04 01:29:49.646327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.646304 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:49.687627 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.687590 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-xks-ca" pod "1f7b38a6-9f62-4105-a347-a9b064d18ce1" (UID: "1f7b38a6-9f62-4105-a347-a9b064d18ce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:49.688003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.687975 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7b38a6-9f62-4105-a347-a9b064d18ce1-kube-api-access-95qdk" pod "1f7b38a6-9f62-4105-a347-a9b064d18ce1" (UID: "1f7b38a6-9f62-4105-a347-a9b064d18ce1"). InnerVolumeSpecName "kube-api-access-95qdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:49.748350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.748316 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95qdk\" (UniqueName: \"kubernetes.io/projected/1f7b38a6-9f62-4105-a347-a9b064d18ce1-kube-api-access-95qdk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:49.748350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.748345 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f7b38a6-9f62-4105-a347-a9b064d18ce1-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:49.895721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.849701 5051 generic.go:356] "Generic (PLEG): container finished" podID="45230ed9-1e0d-4579-a9da-4be008858d09" containerID="2b7d340067d428ddfae56ceb1c465eddb6333d3d278cd6b4d696ba409c9c5375" exitCode=0 Nov 04 01:29:49.895721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.849775 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" event={"ID":"45230ed9-1e0d-4579-a9da-4be008858d09","Type":"ContainerDied","Data":"2b7d340067d428ddfae56ceb1c465eddb6333d3d278cd6b4d696ba409c9c5375"} Nov 04 01:29:49.895721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.851404 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" event={"ID":"05cdea56-609e-4f8e-b99d-37d1e5b7f916","Type":"ContainerStarted","Data":"59bcd37f404a89e8fd3fe9ebf7edb8dfcbce4b412cb901b5f4cd73563ec28589"} Nov 04 01:29:49.895721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.852885 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" event={"ID":"1f7b38a6-9f62-4105-a347-a9b064d18ce1","Type":"ContainerDied","Data":"8a1fe80606a96cdce656794ed9415685d2240fca53b4b8ab28b90e49f4bebaf0"} Nov 04 01:29:49.895721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.852904 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6" Nov 04 01:29:49.895721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:49.852930 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a1fe80606a96cdce656794ed9415685d2240fca53b4b8ab28b90e49f4bebaf0" Nov 04 01:29:50.046161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.045511 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": dial tcp 10.132.0.99:8000: connect: connection refused" Nov 04 01:29:50.046161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.045632 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:29:50.743203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.742766 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:29:50.778028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.777988 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-tmp-dir\") pod \"45230ed9-1e0d-4579-a9da-4be008858d09\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " Nov 04 01:29:50.778746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.778720 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-dshm\") pod \"45230ed9-1e0d-4579-a9da-4be008858d09\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " Nov 04 01:29:50.778977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.778960 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-home\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-home\") pod \"45230ed9-1e0d-4579-a9da-4be008858d09\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " Nov 04 01:29:50.779188 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.779172 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-model-cache\") pod \"45230ed9-1e0d-4579-a9da-4be008858d09\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " Nov 04 01:29:50.779352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.779338 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45230ed9-1e0d-4579-a9da-4be008858d09-tls-certs\" (UniqueName: \"kubernetes.io/secret/45230ed9-1e0d-4579-a9da-4be008858d09-tls-certs\") pod \"45230ed9-1e0d-4579-a9da-4be008858d09\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " Nov 04 01:29:50.779545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.779530 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45230ed9-1e0d-4579-a9da-4be008858d09-kube-api-access-zbnnn\" (UniqueName: \"kubernetes.io/projected/45230ed9-1e0d-4579-a9da-4be008858d09-kube-api-access-zbnnn\") pod \"45230ed9-1e0d-4579-a9da-4be008858d09\" (UID: \"45230ed9-1e0d-4579-a9da-4be008858d09\") " Nov 04 01:29:50.835494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.835409 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-tmp-dir" pod "45230ed9-1e0d-4579-a9da-4be008858d09" (UID: "45230ed9-1e0d-4579-a9da-4be008858d09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.835503 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45230ed9-1e0d-4579-a9da-4be008858d09-kube-api-access-zbnnn" pod "45230ed9-1e0d-4579-a9da-4be008858d09" (UID: "45230ed9-1e0d-4579-a9da-4be008858d09"). InnerVolumeSpecName "kube-api-access-zbnnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.837339 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-home" pod "45230ed9-1e0d-4579-a9da-4be008858d09" (UID: "45230ed9-1e0d-4579-a9da-4be008858d09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.841731 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-model-cache" pod "45230ed9-1e0d-4579-a9da-4be008858d09" (UID: "45230ed9-1e0d-4579-a9da-4be008858d09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.845187 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-dshm" pod "45230ed9-1e0d-4579-a9da-4be008858d09" (UID: "45230ed9-1e0d-4579-a9da-4be008858d09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.849619 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45230ed9-1e0d-4579-a9da-4be008858d09-tls-certs" pod "45230ed9-1e0d-4579-a9da-4be008858d09" (UID: "45230ed9-1e0d-4579-a9da-4be008858d09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.876526 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" event={"ID":"05cdea56-609e-4f8e-b99d-37d1e5b7f916","Type":"ContainerStarted","Data":"0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a"} Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.885406 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.885435 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.885451 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.885623 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45230ed9-1e0d-4579-a9da-4be008858d09-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.885642 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbnnn\" (UniqueName: \"kubernetes.io/projected/45230ed9-1e0d-4579-a9da-4be008858d09-kube-api-access-zbnnn\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.885681 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45230ed9-1e0d-4579-a9da-4be008858d09-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.887608 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" event={"ID":"45230ed9-1e0d-4579-a9da-4be008858d09","Type":"ContainerDied","Data":"b7c24d48e59cda9865cbb1bca850ec0392390f44a350f61c7d0ee4535c423612"} Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.887654 5051 scope.go:122] "RemoveContainer" containerID="2b7d340067d428ddfae56ceb1c465eddb6333d3d278cd6b4d696ba409c9c5375" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.887800 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz" Nov 04 01:29:50.915749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.912072 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podStartSLOduration=4.912052549 podStartE2EDuration="4.912052549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:50.909723815 +0000 UTC m=+1929.881472713" watchObservedRunningTime="2026-11-04 01:29:50.912052549 +0000 UTC m=+1929.883801421" Nov 04 01:29:51.006028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.929844 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz"] Nov 04 01:29:51.006028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:50.937809 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee876a/distinct-w0-ee876a-kserve-667fbd9c7f-qk8cz"] Nov 04 01:29:51.214467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:51.214429 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" path="/var/lib/kubelet/pods/45230ed9-1e0d-4579-a9da-4be008858d09/volumes" Nov 04 01:29:51.418292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:51.418207 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:51.420453 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:51.420414 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 04 01:29:51.420610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:51.420478 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 04 01:29:52.418368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.418333 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 04 01:29:52.418834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.418411 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 04 01:29:52.916183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.915648 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm"] Nov 04 01:29:52.919413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.917978 5051 cpu_manager.go:414] "containerMap: removing container" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" Nov 04 01:29:52.919413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.918009 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" Nov 04 01:29:52.919413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.918057 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1f7b38a6-9f62-4105-a347-a9b064d18ce1" containerName="revoke-keys" Nov 04 01:29:52.919413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.918065 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f7b38a6-9f62-4105-a347-a9b064d18ce1" containerName="revoke-keys" Nov 04 01:29:52.919413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.918287 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="45230ed9-1e0d-4579-a9da-4be008858d09" containerName="main" Nov 04 01:29:52.919413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:52.918311 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f7b38a6-9f62-4105-a347-a9b064d18ce1" containerName="revoke-keys" Nov 04 01:29:53.083150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.083076 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm"] Nov 04 01:29:53.083342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.083249 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.198986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.198892 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e34ad9-7fee-457e-9a62-98badb115fb0-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.199210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.199006 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.199210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.199039 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.199210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.199082 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.199384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.199253 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.199384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.199318 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.199508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.199391 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd6gb\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-kube-api-access-wd6gb\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.199508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.199473 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.199615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.199536 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.301287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301254 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.301502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301326 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6gb\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-kube-api-access-wd6gb\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.301502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301412 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.301640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301521 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.301640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301579 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e34ad9-7fee-457e-9a62-98badb115fb0-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301729 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301760 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301780 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.301865 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.302020 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.302052 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.302223 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.302457 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.302534 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e34ad9-7fee-457e-9a62-98badb115fb0-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.304164 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.304813 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.309859 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6gb\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-kube-api-access-wd6gb\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.328090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.311969 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:53.420090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.418889 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 04 01:29:53.420090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.418949 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 04 01:29:53.536899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:53.536636 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:54.033386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:54.033272 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm"] Nov 04 01:29:54.039291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:54.039235 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:54.039404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:54.039326 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:54.039404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:54.039375 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:29:54.592883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:54.586549 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 04 01:29:54.592883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:54.586614 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 04 01:29:55.037768 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:55.037730 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" event={"ID":"70e34ad9-7fee-457e-9a62-98badb115fb0","Type":"ContainerStarted","Data":"cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e"} Nov 04 01:29:55.039644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:55.039616 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" event={"ID":"70e34ad9-7fee-457e-9a62-98badb115fb0","Type":"ContainerStarted","Data":"db2b8d31937d268f9957db1cd1aa4690a3902c98b4d3a4da92bb9cae54ccbb58"} Nov 04 01:29:55.087145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:55.079900 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podStartSLOduration=3.079880271 podStartE2EDuration="3.079880271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:29:55.070568193 +0000 UTC m=+1934.042317068" watchObservedRunningTime="2026-11-04 01:29:55.079880271 +0000 UTC m=+1934.051629144" Nov 04 01:29:55.431220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:55.424970 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.431220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:55.425032 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 04 01:29:55.539168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:55.538286 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:29:55.539869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:55.539846 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 04 01:29:55.569909 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:55.539902 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 04 01:29:56.464145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:56.459256 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 04 01:29:56.464145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:56.459323 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 04 01:29:56.542155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:56.540095 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 04 01:29:56.542155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:56.540226 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 04 01:29:57.426591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:57.426162 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.426591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:57.426218 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 04 01:29:57.573254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:57.553912 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 04 01:29:57.573254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:57.553976 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 04 01:29:58.421552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:58.421272 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 04 01:29:58.421552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:58.421339 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 04 01:29:58.543943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:58.543473 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 04 01:29:58.543943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:58.543539 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 04 01:29:59.478322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:59.476351 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:59.478322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:59.478156 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:59.479103 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:59.479039 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:29:59.538160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:59.538103 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 04 01:29:59.538344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:29:59.538180 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 04 01:30:00.101285 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.101206 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj"] Nov 04 01:30:00.180354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.180303 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3cc47b3-p8ptj"] Nov 04 01:30:00.212883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.212848 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn"] Nov 04 01:30:00.352190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.352059 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj"] Nov 04 01:30:00.394785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.361715 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:00.394785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.392456 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsd7\" (UniqueName: \"kubernetes.io/projected/4f36bcd3-2829-4dad-a069-1cb09f599e0c-kube-api-access-mcsd7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn\" (UID: \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:00.394785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.392525 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36bcd3-2829-4dad-a069-1cb09f599e0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn\" (UID: \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:00.455357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.405355 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:30:00.455357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.405542 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:30:00.455357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.407677 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:30:00.477177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.457966 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827"] Nov 04 01:30:00.477177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.465240 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:00.507154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.504357 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsd7\" (UniqueName: \"kubernetes.io/projected/4f36bcd3-2829-4dad-a069-1cb09f599e0c-kube-api-access-mcsd7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn\" (UID: \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:00.507154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.504442 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36bcd3-2829-4dad-a069-1cb09f599e0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn\" (UID: \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:00.507154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.504889 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj\" (UID: \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:00.507154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.504988 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2gfw\" (UniqueName: \"kubernetes.io/projected/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-kube-api-access-n2gfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj\" (UID: \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:00.545841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.543946 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 04 01:30:00.545841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.544241 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 04 01:30:00.624163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.624050 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36bcd3-2829-4dad-a069-1cb09f599e0c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn\" (UID: \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:00.637740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.637705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj\" (UID: \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:00.637912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.637836 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2gfw\" (UniqueName: \"kubernetes.io/projected/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-kube-api-access-n2gfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj\" (UID: \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:00.638967 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.638932 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj\" (UID: \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:00.644571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.644549 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn"] Nov 04 01:30:00.644730 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.644712 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d"] Nov 04 01:30:00.648011 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.647992 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:00.714186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.709045 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsd7\" (UniqueName: \"kubernetes.io/projected/4f36bcd3-2829-4dad-a069-1cb09f599e0c-kube-api-access-mcsd7\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn\" (UID: \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:00.721851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.720423 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2gfw\" (UniqueName: \"kubernetes.io/projected/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-kube-api-access-n2gfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj\" (UID: \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:00.723645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.723470 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s"] Nov 04 01:30:00.725541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.724342 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:00.740848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.740344 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l89lj\" (UniqueName: \"kubernetes.io/projected/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-kube-api-access-l89lj\") pod \"maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827\" (UID: \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:00.740848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.740439 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827\" (UID: \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:00.803065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.803007 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj"] Nov 04 01:30:00.803253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.803233 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957"] Nov 04 01:30:00.803348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.803106 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:00.843165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.843096 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l89lj\" (UniqueName: \"kubernetes.io/projected/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-kube-api-access-l89lj\") pod \"maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827\" (UID: \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:00.843364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.843214 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dqk\" (UniqueName: \"kubernetes.io/projected/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-kube-api-access-26dqk\") pod \"maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s\" (UID: \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:00.843364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.843256 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827\" (UID: \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:00.843583 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.843559 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f066d9df-2697-461e-92db-66abc15c1788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d\" (UID: \"f066d9df-2697-461e-92db-66abc15c1788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:00.843656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.843624 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7bfn\" (UniqueName: \"kubernetes.io/projected/f066d9df-2697-461e-92db-66abc15c1788-kube-api-access-q7bfn\") pod \"maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d\" (UID: \"f066d9df-2697-461e-92db-66abc15c1788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:00.843729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.843703 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s\" (UID: \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:00.890088 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.843988 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827\" (UID: \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:00.890088 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.884166 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l89lj\" (UniqueName: \"kubernetes.io/projected/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-kube-api-access-l89lj\") pod \"maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827\" (UID: \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:00.890328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.890251 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j"] Nov 04 01:30:00.890391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.890376 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:00.941209 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.941178 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:00.945298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.945272 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7m6\" (UniqueName: \"kubernetes.io/projected/000eb04e-a38b-424e-a668-feae478b052f-kube-api-access-wb7m6\") pod \"maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957\" (UID: \"000eb04e-a38b-424e-a668-feae478b052f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:00.945426 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.945315 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f066d9df-2697-461e-92db-66abc15c1788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d\" (UID: \"f066d9df-2697-461e-92db-66abc15c1788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:00.945426 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.945400 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7bfn\" (UniqueName: \"kubernetes.io/projected/f066d9df-2697-461e-92db-66abc15c1788-kube-api-access-q7bfn\") pod \"maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d\" (UID: \"f066d9df-2697-461e-92db-66abc15c1788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:00.945505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.945453 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s\" (UID: \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:00.945723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.945705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dqk\" (UniqueName: \"kubernetes.io/projected/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-kube-api-access-26dqk\") pod \"maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s\" (UID: \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:00.945801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.945785 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000eb04e-a38b-424e-a668-feae478b052f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957\" (UID: \"000eb04e-a38b-424e-a668-feae478b052f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:00.948936 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:00.948916 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:01.007880 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.007840 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f066d9df-2697-461e-92db-66abc15c1788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d\" (UID: \"f066d9df-2697-461e-92db-66abc15c1788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:01.008285 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.008254 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s\" (UID: \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:01.008423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.008315 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv"] Nov 04 01:30:01.008764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.008434 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:01.047780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.047753 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn844\" (UniqueName: \"kubernetes.io/projected/9027ba5f-d082-4126-9708-82766ac7e612-kube-api-access-rn844\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j\" (UID: \"9027ba5f-d082-4126-9708-82766ac7e612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:01.048063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.048043 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000eb04e-a38b-424e-a668-feae478b052f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957\" (UID: \"000eb04e-a38b-424e-a668-feae478b052f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:01.048171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.048099 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9027ba5f-d082-4126-9708-82766ac7e612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j\" (UID: \"9027ba5f-d082-4126-9708-82766ac7e612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:01.048308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.048289 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7m6\" (UniqueName: \"kubernetes.io/projected/000eb04e-a38b-424e-a668-feae478b052f-kube-api-access-wb7m6\") pod \"maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957\" (UID: \"000eb04e-a38b-424e-a668-feae478b052f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:01.070355 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.069736 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dqk\" (UniqueName: \"kubernetes.io/projected/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-kube-api-access-26dqk\") pod \"maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s\" (UID: \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:01.070355 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.070315 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7bfn\" (UniqueName: \"kubernetes.io/projected/f066d9df-2697-461e-92db-66abc15c1788-kube-api-access-q7bfn\") pod \"maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d\" (UID: \"f066d9df-2697-461e-92db-66abc15c1788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:01.072857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.072820 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000eb04e-a38b-424e-a668-feae478b052f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957\" (UID: \"000eb04e-a38b-424e-a668-feae478b052f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:01.083099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.083070 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7m6\" (UniqueName: \"kubernetes.io/projected/000eb04e-a38b-424e-a668-feae478b052f-kube-api-access-wb7m6\") pod \"maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957\" (UID: \"000eb04e-a38b-424e-a668-feae478b052f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:01.104485 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.104460 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:01.203149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.180476 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:01.203149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.181315 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:01.203149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.196246 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn844\" (UniqueName: \"kubernetes.io/projected/9027ba5f-d082-4126-9708-82766ac7e612-kube-api-access-rn844\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j\" (UID: \"9027ba5f-d082-4126-9708-82766ac7e612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:01.203149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.196617 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9027ba5f-d082-4126-9708-82766ac7e612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j\" (UID: \"9027ba5f-d082-4126-9708-82766ac7e612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:01.203149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.197831 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9027ba5f-d082-4126-9708-82766ac7e612-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j\" (UID: \"9027ba5f-d082-4126-9708-82766ac7e612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:01.237798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.237703 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn844\" (UniqueName: \"kubernetes.io/projected/9027ba5f-d082-4126-9708-82766ac7e612-kube-api-access-rn844\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j\" (UID: \"9027ba5f-d082-4126-9708-82766ac7e612\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:01.265665 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.265625 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:01.288184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.288152 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:01.367484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.337543 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:01.421320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.407010 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cp75\" (UniqueName: \"kubernetes.io/projected/b5f7a55e-0d37-4aeb-9888-32c89c718adf-kube-api-access-6cp75\") pod \"maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv\" (UID: \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:01.421320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.407105 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f7a55e-0d37-4aeb-9888-32c89c718adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv\" (UID: \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:01.515942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.515909 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cp75\" (UniqueName: \"kubernetes.io/projected/b5f7a55e-0d37-4aeb-9888-32c89c718adf-kube-api-access-6cp75\") pod \"maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv\" (UID: \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:01.516370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.515992 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f7a55e-0d37-4aeb-9888-32c89c718adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv\" (UID: \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:01.519330 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.519151 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3" path="/var/lib/kubelet/pods/a14f96d4-9de5-4bfc-beef-9e7f76b7b4d3/volumes" Nov 04 01:30:01.519868 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.519839 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f7a55e-0d37-4aeb-9888-32c89c718adf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv\" (UID: \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.530613 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cp75\" (UniqueName: \"kubernetes.io/projected/b5f7a55e-0d37-4aeb-9888-32c89c718adf-kube-api-access-6cp75\") pod \"maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv\" (UID: \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.539022 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.539063 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.609908 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9"] Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.714461 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.724581 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8"] Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.727576 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.948110 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ngs\" (UniqueName: \"kubernetes.io/projected/1d81258f-8ed3-4153-9d49-41656cb054fe-kube-api-access-f5ngs\") pod \"maas-api-key-cleanup-29895930-dp6m9\" (UID: \"1d81258f-8ed3-4153-9d49-41656cb054fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:01.993005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:01.948275 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81258f-8ed3-4153-9d49-41656cb054fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-dp6m9\" (UID: \"1d81258f-8ed3-4153-9d49-41656cb054fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:02.009971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.007416 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827"] Nov 04 01:30:02.009971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.007477 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d"] Nov 04 01:30:02.009971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.007508 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s"] Nov 04 01:30:02.009971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.007536 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957"] Nov 04 01:30:02.009971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.007562 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j"] Nov 04 01:30:02.009971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.007592 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv"] Nov 04 01:30:02.009971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.007621 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9"] Nov 04 01:30:02.009971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.007648 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8"] Nov 04 01:30:02.032831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.013473 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:02.181789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.135785 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134be8d-cd30-414f-b35c-0ce688b2184f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8\" (UID: \"4134be8d-cd30-414f-b35c-0ce688b2184f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:02.181789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.135845 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5bx\" (UniqueName: \"kubernetes.io/projected/4134be8d-cd30-414f-b35c-0ce688b2184f-kube-api-access-hw5bx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8\" (UID: \"4134be8d-cd30-414f-b35c-0ce688b2184f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:02.181789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.135910 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ngs\" (UniqueName: \"kubernetes.io/projected/1d81258f-8ed3-4153-9d49-41656cb054fe-kube-api-access-f5ngs\") pod \"maas-api-key-cleanup-29895930-dp6m9\" (UID: \"1d81258f-8ed3-4153-9d49-41656cb054fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:02.181789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.136047 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81258f-8ed3-4153-9d49-41656cb054fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-dp6m9\" (UID: \"1d81258f-8ed3-4153-9d49-41656cb054fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:02.227016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.226931 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81258f-8ed3-4153-9d49-41656cb054fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895930-dp6m9\" (UID: \"1d81258f-8ed3-4153-9d49-41656cb054fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:02.302288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.284595 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134be8d-cd30-414f-b35c-0ce688b2184f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8\" (UID: \"4134be8d-cd30-414f-b35c-0ce688b2184f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:02.302288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.284661 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5bx\" (UniqueName: \"kubernetes.io/projected/4134be8d-cd30-414f-b35c-0ce688b2184f-kube-api-access-hw5bx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8\" (UID: \"4134be8d-cd30-414f-b35c-0ce688b2184f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:02.302288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.286503 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134be8d-cd30-414f-b35c-0ce688b2184f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8\" (UID: \"4134be8d-cd30-414f-b35c-0ce688b2184f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:02.506171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.352585 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ngs\" (UniqueName: \"kubernetes.io/projected/1d81258f-8ed3-4153-9d49-41656cb054fe-kube-api-access-f5ngs\") pod \"maas-api-key-cleanup-29895930-dp6m9\" (UID: \"1d81258f-8ed3-4153-9d49-41656cb054fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:02.589967 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.569201 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:02.790514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.772546 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:30:02.790514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.773689 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:30:02.921250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.914222 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:30:02.942609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.942534 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5bx\" (UniqueName: \"kubernetes.io/projected/4134be8d-cd30-414f-b35c-0ce688b2184f-kube-api-access-hw5bx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8\" (UID: \"4134be8d-cd30-414f-b35c-0ce688b2184f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:02.973412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:02.973254 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:03.131656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.053363 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5"] Nov 04 01:30:03.131656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.121102 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5403a6-bk8h5"] Nov 04 01:30:03.184180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.159970 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957"] Nov 04 01:30:03.218202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.207808 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220ad717-fb38-4ec1-aaa3-6dc341f1fa3a" path="/var/lib/kubelet/pods/220ad717-fb38-4ec1-aaa3-6dc341f1fa3a/volumes" Nov 04 01:30:03.244525 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:30:03.237398 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000eb04e_a38b_424e_a668_feae478b052f.slice/crio-de7afdf43d080f2d786f54dcd04b1d4da3e74a82bc3da93296dc1c588257faab WatchSource:0}: Error finding container de7afdf43d080f2d786f54dcd04b1d4da3e74a82bc3da93296dc1c588257faab: Status 404 returned error can't find the container with id de7afdf43d080f2d786f54dcd04b1d4da3e74a82bc3da93296dc1c588257faab Nov 04 01:30:03.444715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.436105 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:03.444715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.436395 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:03.641917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.610597 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9"] Nov 04 01:30:03.669802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.654030 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn"] Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:30:03.694990 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fd0185_4925_4a4b_9c7c_67678d2da4b8.slice/crio-a5bb10b6d2b96f8603edc6f21c0f4f8a1c2505f2883d1c866e93d3bf4bc12062 WatchSource:0}: Error finding container a5bb10b6d2b96f8603edc6f21c0f4f8a1c2505f2883d1c866e93d3bf4bc12062: Status 404 returned error can't find the container with id a5bb10b6d2b96f8603edc6f21c0f4f8a1c2505f2883d1c866e93d3bf4bc12062 Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.695163 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8"] Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.702903 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827"] Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.710268 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv"] Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.715166 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s"] Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.720107 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d"] Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:30:03.727652 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4134be8d_cd30_414f_b35c_0ce688b2184f.slice/crio-85e4a8cd841759b5f6780fc3ea9312a83385ec1d23b319690bfcf99c98f7921a WatchSource:0}: Error finding container 85e4a8cd841759b5f6780fc3ea9312a83385ec1d23b319690bfcf99c98f7921a: Status 404 returned error can't find the container with id 85e4a8cd841759b5f6780fc3ea9312a83385ec1d23b319690bfcf99c98f7921a Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.729991 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj"] Nov 04 01:30:03.745580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:03.737520 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j"] Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.070337 5051 generic.go:356] "Generic (PLEG): container finished" podID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerID="2f0ffce8ce0c12d65173062cd7d74c4a1fccde75ebb2da2355d7669843e7ab40" exitCode=0 Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.070431 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" event={"ID":"3631a39e-5bb5-4de8-9fa2-2830f3f00a94","Type":"ContainerDied","Data":"2f0ffce8ce0c12d65173062cd7d74c4a1fccde75ebb2da2355d7669843e7ab40"} Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.070962 5051 scope.go:122] "RemoveContainer" containerID="2f0ffce8ce0c12d65173062cd7d74c4a1fccde75ebb2da2355d7669843e7ab40" Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.073009 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" event={"ID":"4134be8d-cd30-414f-b35c-0ce688b2184f","Type":"ContainerStarted","Data":"85e4a8cd841759b5f6780fc3ea9312a83385ec1d23b319690bfcf99c98f7921a"} Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.074766 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" event={"ID":"f066d9df-2697-461e-92db-66abc15c1788","Type":"ContainerStarted","Data":"a3f1d64f0f9b9e756a30fa13c84fcbb1a2bf717032f50e9c4e80d2dde0136823"} Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.076805 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" event={"ID":"9027ba5f-d082-4126-9708-82766ac7e612","Type":"ContainerStarted","Data":"60983a393a58f75eaec872ce4e4acb5ac21a72af8fb16ab5414d1d2fae845ab5"} Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.078586 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" event={"ID":"b0fd0185-4925-4a4b-9c7c-67678d2da4b8","Type":"ContainerStarted","Data":"a5bb10b6d2b96f8603edc6f21c0f4f8a1c2505f2883d1c866e93d3bf4bc12062"} Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.080649 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" event={"ID":"000eb04e-a38b-424e-a668-feae478b052f","Type":"ContainerStarted","Data":"de7afdf43d080f2d786f54dcd04b1d4da3e74a82bc3da93296dc1c588257faab"} Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.082483 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" event={"ID":"f4245e28-a7cf-4df5-a41f-73a0da0f78fa","Type":"ContainerStarted","Data":"ee476971b30aaa7779a4cfba91a105a08d2fa7eb91b2b471976e22a435e9e0f1"} Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.084831 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" event={"ID":"b5f7a55e-0d37-4aeb-9888-32c89c718adf","Type":"ContainerStarted","Data":"f18ceb228a83647621cf29412918637ce8543faeb5774eab7f26653db1b78990"} Nov 04 01:30:04.091770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.086485 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" event={"ID":"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f","Type":"ContainerStarted","Data":"cb0c8c6de1026934b8a7e320609576d456d041559e36d8714c587a1a39b3cadb"} Nov 04 01:30:04.179762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.094689 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" event={"ID":"4f36bcd3-2829-4dad-a069-1cb09f599e0c","Type":"ContainerStarted","Data":"dbcf4ea1cc6a81ecf1f87fe3388b2b326219638c75aa93d21227f4b98a260f12"} Nov 04 01:30:04.179762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.098900 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" event={"ID":"1d81258f-8ed3-4153-9d49-41656cb054fe","Type":"ContainerStarted","Data":"d5f5815ab0df2f8501cc5f6df51466b58d6dcb6e4025dc487902db66d3426c48"} Nov 04 01:30:04.394262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.238416 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:04.394262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.238503 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:04.394262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.238538 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:04.509747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.509023 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:30:04.509747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:04.509071 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:04.973428 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:04.973392 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:30:05.111189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:05.111042 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" event={"ID":"000eb04e-a38b-424e-a668-feae478b052f","Type":"ContainerStarted","Data":"1fb5fed23c66ac5aba0f1439c50d0441cc25a9e291b83c501b0cbc31e14e4e0c"} Nov 04 01:30:05.113542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:05.113512 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" event={"ID":"b5f7a55e-0d37-4aeb-9888-32c89c718adf","Type":"ContainerStarted","Data":"cda3d27eb5f7399d6157c267b82e467e128a2826111d80ae141f986e3076051a"} Nov 04 01:30:05.115535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:05.115507 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" event={"ID":"4f36bcd3-2829-4dad-a069-1cb09f599e0c","Type":"ContainerStarted","Data":"a02b746271ed4bf37ffdc9c11919f7d9cab12343e3a54b3f3d9ffcf336606f20"} Nov 04 01:30:05.117712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:05.117651 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" event={"ID":"1d81258f-8ed3-4153-9d49-41656cb054fe","Type":"ContainerStarted","Data":"0fa0f2168f9e1df0cc74eda1c35bcc6ce017b696d71da67bafd776295029df23"} Nov 04 01:30:05.144329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:05.144279 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" podStartSLOduration=5.144267559 podStartE2EDuration="5.144267559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:05.139638419 +0000 UTC m=+1944.111387301" watchObservedRunningTime="2026-11-04 01:30:05.144267559 +0000 UTC m=+1944.116016474" Nov 04 01:30:06.129981 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.129949 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" event={"ID":"3631a39e-5bb5-4de8-9fa2-2830f3f00a94","Type":"ContainerStarted","Data":"fe29a4a9d21efc11ad44e5b25ae142024a216371d5ddc570981711232098ddf0"} Nov 04 01:30:06.131788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.131760 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" event={"ID":"4134be8d-cd30-414f-b35c-0ce688b2184f","Type":"ContainerStarted","Data":"6ec6d26f027f259379edd0633bf6c2305ba434becc61bbd42cc52d93dff47e01"} Nov 04 01:30:06.133877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.133850 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" event={"ID":"f066d9df-2697-461e-92db-66abc15c1788","Type":"ContainerStarted","Data":"64845c4108b2b3102b98dd7ead4ace893bb96148f358a6a0e3a0387869aedb58"} Nov 04 01:30:06.136993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.136966 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" event={"ID":"9027ba5f-d082-4126-9708-82766ac7e612","Type":"ContainerStarted","Data":"bdad473437a8b3ff53222fa757c05eea5c51ef09ed755941c977b72f82871acc"} Nov 04 01:30:06.139314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.139292 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" event={"ID":"b0fd0185-4925-4a4b-9c7c-67678d2da4b8","Type":"ContainerStarted","Data":"48f084feb07de17bf84962c99f39b16d246763b62bdf152c960ebecf6274edb5"} Nov 04 01:30:06.141012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.140986 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" event={"ID":"f4245e28-a7cf-4df5-a41f-73a0da0f78fa","Type":"ContainerStarted","Data":"b931770a90bbf816116a542d26769778ccd92df4253a0c987978adc47519b4d2"} Nov 04 01:30:06.143003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.142977 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" event={"ID":"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f","Type":"ContainerStarted","Data":"b4d8afdc04a7c87e8c4197fa47428ed8a8803a6338f76df0fac5b42e97be7049"} Nov 04 01:30:06.173911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.173857 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" podStartSLOduration=6.173840516 podStartE2EDuration="6.173840516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:05.19264736 +0000 UTC m=+1944.164396242" watchObservedRunningTime="2026-11-04 01:30:06.173840516 +0000 UTC m=+1945.145589388" Nov 04 01:30:06.188055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.188002 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" podStartSLOduration=6.187987441 podStartE2EDuration="6.187987441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.183076126 +0000 UTC m=+1945.154824998" watchObservedRunningTime="2026-11-04 01:30:06.187987441 +0000 UTC m=+1945.159736314" Nov 04 01:30:06.300931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.203240 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" podStartSLOduration=6.203230135 podStartE2EDuration="6.203230135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.201948103 +0000 UTC m=+1945.173696973" watchObservedRunningTime="2026-11-04 01:30:06.203230135 +0000 UTC m=+1945.174979007" Nov 04 01:30:06.300931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.229314 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" podStartSLOduration=6.229302071 podStartE2EDuration="6.229302071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.224501985 +0000 UTC m=+1945.196250856" watchObservedRunningTime="2026-11-04 01:30:06.229302071 +0000 UTC m=+1945.201050943" Nov 04 01:30:06.300931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.245618 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" podStartSLOduration=6.245601487 podStartE2EDuration="6.245601487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.244573612 +0000 UTC m=+1945.216322482" watchObservedRunningTime="2026-11-04 01:30:06.245601487 +0000 UTC m=+1945.217350358" Nov 04 01:30:06.300931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.269589 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" podStartSLOduration=6.269577627 podStartE2EDuration="6.269577627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.267055447 +0000 UTC m=+1945.238804319" watchObservedRunningTime="2026-11-04 01:30:06.269577627 +0000 UTC m=+1945.241326498" Nov 04 01:30:06.300931 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.293281 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" podStartSLOduration=6.293270179 podStartE2EDuration="6.293270179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.287774547 +0000 UTC m=+1945.259523417" watchObservedRunningTime="2026-11-04 01:30:06.293270179 +0000 UTC m=+1945.265019048" Nov 04 01:30:06.389482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.317782 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" podStartSLOduration=6.31776796 podStartE2EDuration="6.31776796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.316047875 +0000 UTC m=+1945.287796747" watchObservedRunningTime="2026-11-04 01:30:06.31776796 +0000 UTC m=+1945.289516834" Nov 04 01:30:06.389482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.343995 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" podStartSLOduration=6.343983232 podStartE2EDuration="6.343983232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:06.340662118 +0000 UTC m=+1945.312411012" watchObservedRunningTime="2026-11-04 01:30:06.343983232 +0000 UTC m=+1945.315732102" Nov 04 01:30:06.589151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.580893 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:30:06.589151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.581075 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:06.589151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:06.581135 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:07.846082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.584709 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:07.846082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.584766 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:07.935180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.935131 5051 generic.go:356] "Generic (PLEG): container finished" podID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerID="48f084feb07de17bf84962c99f39b16d246763b62bdf152c960ebecf6274edb5" exitCode=60 Nov 04 01:30:07.936073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.936042 5051 scope.go:122] "RemoveContainer" containerID="48f084feb07de17bf84962c99f39b16d246763b62bdf152c960ebecf6274edb5" Nov 04 01:30:07.939315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.938657 5051 generic.go:356] "Generic (PLEG): container finished" podID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerID="b931770a90bbf816116a542d26769778ccd92df4253a0c987978adc47519b4d2" exitCode=60 Nov 04 01:30:07.939400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.939312 5051 scope.go:122] "RemoveContainer" containerID="b931770a90bbf816116a542d26769778ccd92df4253a0c987978adc47519b4d2" Nov 04 01:30:07.944013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.943818 5051 generic.go:356] "Generic (PLEG): container finished" podID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerID="cda3d27eb5f7399d6157c267b82e467e128a2826111d80ae141f986e3076051a" exitCode=60 Nov 04 01:30:07.947457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.947301 5051 generic.go:356] "Generic (PLEG): container finished" podID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerID="0fa0f2168f9e1df0cc74eda1c35bcc6ce017b696d71da67bafd776295029df23" exitCode=60 Nov 04 01:30:07.972190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.963880 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" event={"ID":"b0fd0185-4925-4a4b-9c7c-67678d2da4b8","Type":"ContainerDied","Data":"48f084feb07de17bf84962c99f39b16d246763b62bdf152c960ebecf6274edb5"} Nov 04 01:30:07.972190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.963950 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" event={"ID":"f4245e28-a7cf-4df5-a41f-73a0da0f78fa","Type":"ContainerDied","Data":"b931770a90bbf816116a542d26769778ccd92df4253a0c987978adc47519b4d2"} Nov 04 01:30:07.972190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.963981 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" event={"ID":"b5f7a55e-0d37-4aeb-9888-32c89c718adf","Type":"ContainerDied","Data":"cda3d27eb5f7399d6157c267b82e467e128a2826111d80ae141f986e3076051a"} Nov 04 01:30:07.972190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.964009 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" event={"ID":"1d81258f-8ed3-4153-9d49-41656cb054fe","Type":"ContainerDied","Data":"0fa0f2168f9e1df0cc74eda1c35bcc6ce017b696d71da67bafd776295029df23"} Nov 04 01:30:07.989968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.978688 5051 scope.go:122] "RemoveContainer" containerID="0fa0f2168f9e1df0cc74eda1c35bcc6ce017b696d71da67bafd776295029df23" Nov 04 01:30:07.989968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:07.986746 5051 scope.go:122] "RemoveContainer" containerID="cda3d27eb5f7399d6157c267b82e467e128a2826111d80ae141f986e3076051a" Nov 04 01:30:08.597145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:08.590948 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:08.597145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:08.591034 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.003922 5051 generic.go:356] "Generic (PLEG): container finished" podID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerID="6ec6d26f027f259379edd0633bf6c2305ba434becc61bbd42cc52d93dff47e01" exitCode=60 Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.004042 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" event={"ID":"4134be8d-cd30-414f-b35c-0ce688b2184f","Type":"ContainerDied","Data":"6ec6d26f027f259379edd0633bf6c2305ba434becc61bbd42cc52d93dff47e01"} Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.006211 5051 generic.go:356] "Generic (PLEG): container finished" podID="f066d9df-2697-461e-92db-66abc15c1788" containerID="64845c4108b2b3102b98dd7ead4ace893bb96148f358a6a0e3a0387869aedb58" exitCode=60 Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.006315 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" event={"ID":"f066d9df-2697-461e-92db-66abc15c1788","Type":"ContainerDied","Data":"64845c4108b2b3102b98dd7ead4ace893bb96148f358a6a0e3a0387869aedb58"} Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.006634 5051 scope.go:122] "RemoveContainer" containerID="6ec6d26f027f259379edd0633bf6c2305ba434becc61bbd42cc52d93dff47e01" Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.006897 5051 scope.go:122] "RemoveContainer" containerID="64845c4108b2b3102b98dd7ead4ace893bb96148f358a6a0e3a0387869aedb58" Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.008664 5051 generic.go:356] "Generic (PLEG): container finished" podID="9027ba5f-d082-4126-9708-82766ac7e612" containerID="bdad473437a8b3ff53222fa757c05eea5c51ef09ed755941c977b72f82871acc" exitCode=60 Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.008814 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" event={"ID":"9027ba5f-d082-4126-9708-82766ac7e612","Type":"ContainerDied","Data":"bdad473437a8b3ff53222fa757c05eea5c51ef09ed755941c977b72f82871acc"} Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.009225 5051 scope.go:122] "RemoveContainer" containerID="bdad473437a8b3ff53222fa757c05eea5c51ef09ed755941c977b72f82871acc" Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.011575 5051 generic.go:356] "Generic (PLEG): container finished" podID="000eb04e-a38b-424e-a668-feae478b052f" containerID="1fb5fed23c66ac5aba0f1439c50d0441cc25a9e291b83c501b0cbc31e14e4e0c" exitCode=60 Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.011640 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" event={"ID":"000eb04e-a38b-424e-a668-feae478b052f","Type":"ContainerDied","Data":"1fb5fed23c66ac5aba0f1439c50d0441cc25a9e291b83c501b0cbc31e14e4e0c"} Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.013181 5051 generic.go:356] "Generic (PLEG): container finished" podID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerID="b4d8afdc04a7c87e8c4197fa47428ed8a8803a6338f76df0fac5b42e97be7049" exitCode=60 Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.013259 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" event={"ID":"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f","Type":"ContainerDied","Data":"b4d8afdc04a7c87e8c4197fa47428ed8a8803a6338f76df0fac5b42e97be7049"} Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.013606 5051 scope.go:122] "RemoveContainer" containerID="b4d8afdc04a7c87e8c4197fa47428ed8a8803a6338f76df0fac5b42e97be7049" Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.013725 5051 scope.go:122] "RemoveContainer" containerID="1fb5fed23c66ac5aba0f1439c50d0441cc25a9e291b83c501b0cbc31e14e4e0c" Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.016492 5051 generic.go:356] "Generic (PLEG): container finished" podID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerID="a02b746271ed4bf37ffdc9c11919f7d9cab12343e3a54b3f3d9ffcf336606f20" exitCode=60 Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.016527 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" event={"ID":"4f36bcd3-2829-4dad-a069-1cb09f599e0c","Type":"ContainerDied","Data":"a02b746271ed4bf37ffdc9c11919f7d9cab12343e3a54b3f3d9ffcf336606f20"} Nov 04 01:30:09.063628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.016799 5051 scope.go:122] "RemoveContainer" containerID="a02b746271ed4bf37ffdc9c11919f7d9cab12343e3a54b3f3d9ffcf336606f20" Nov 04 01:30:09.588597 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.579597 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:09.588597 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:09.579654 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:10.580614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.580587 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:10.581295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.581262 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:10.608395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.608308 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" event={"ID":"b0fd0185-4925-4a4b-9c7c-67678d2da4b8","Type":"ContainerStarted","Data":"08650e245d31f207c73e69eccd6052c841b3125c6438157ba51772e3b1645da0"} Nov 04 01:30:10.617672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.617640 5051 generic.go:356] "Generic (PLEG): container finished" podID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerID="e86ed5fb74795c1fedb71f99dc87ce27eb6a2a9697a9b3c1262a0a1770116ec5" exitCode=60 Nov 04 01:30:10.617989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.617965 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" event={"ID":"f4245e28-a7cf-4df5-a41f-73a0da0f78fa","Type":"ContainerDied","Data":"e86ed5fb74795c1fedb71f99dc87ce27eb6a2a9697a9b3c1262a0a1770116ec5"} Nov 04 01:30:10.618182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.618167 5051 scope.go:122] "RemoveContainer" containerID="b931770a90bbf816116a542d26769778ccd92df4253a0c987978adc47519b4d2" Nov 04 01:30:10.624045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.620278 5051 scope.go:122] "RemoveContainer" containerID="e86ed5fb74795c1fedb71f99dc87ce27eb6a2a9697a9b3c1262a0a1770116ec5" Nov 04 01:30:10.624045 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:10.620757 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj_odh-ai-gateway-infra(f4245e28-a7cf-4df5-a41f-73a0da0f78fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" Nov 04 01:30:10.630328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.629147 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" event={"ID":"b5f7a55e-0d37-4aeb-9888-32c89c718adf","Type":"ContainerStarted","Data":"5425c58dd85e63173a3b224cd3c98c5b70e9deffdb824bde134b249d4ad55135"} Nov 04 01:30:10.630328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.629532 5051 scope.go:122] "RemoveContainer" containerID="5425c58dd85e63173a3b224cd3c98c5b70e9deffdb824bde134b249d4ad55135" Nov 04 01:30:10.630328 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:10.629934 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv_odh-ai-gateway-infra(b5f7a55e-0d37-4aeb-9888-32c89c718adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" Nov 04 01:30:10.635139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.635096 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" event={"ID":"1d81258f-8ed3-4153-9d49-41656cb054fe","Type":"ContainerStarted","Data":"648c71627331c8b37e2cff6f60d962244f7c4f6a79f793515bd027f8c58dde6f"} Nov 04 01:30:10.636427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:10.636372 5051 scope.go:122] "RemoveContainer" containerID="648c71627331c8b37e2cff6f60d962244f7c4f6a79f793515bd027f8c58dde6f" Nov 04 01:30:10.636738 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:10.636716 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-dp6m9_odh-ai-gateway-infra(1d81258f-8ed3-4153-9d49-41656cb054fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" Nov 04 01:30:11.584097 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.583638 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:11.584097 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.583703 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:11.692156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.691776 5051 generic.go:356] "Generic (PLEG): container finished" podID="f066d9df-2697-461e-92db-66abc15c1788" containerID="d874347e421aade3534a9cdc4bd52c78cca20ec815987eeaa1a0cfd0987710a7" exitCode=60 Nov 04 01:30:11.692156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.691963 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" event={"ID":"f066d9df-2697-461e-92db-66abc15c1788","Type":"ContainerDied","Data":"d874347e421aade3534a9cdc4bd52c78cca20ec815987eeaa1a0cfd0987710a7"} Nov 04 01:30:11.692156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.692026 5051 scope.go:122] "RemoveContainer" containerID="64845c4108b2b3102b98dd7ead4ace893bb96148f358a6a0e3a0387869aedb58" Nov 04 01:30:11.699599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.699568 5051 scope.go:122] "RemoveContainer" containerID="d874347e421aade3534a9cdc4bd52c78cca20ec815987eeaa1a0cfd0987710a7" Nov 04 01:30:11.848882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.811388 5051 generic.go:356] "Generic (PLEG): container finished" podID="9027ba5f-d082-4126-9708-82766ac7e612" containerID="890cd1373fa0f18d52cff292516d425f191acfb781fc1e93cc5887fa57a043d9" exitCode=60 Nov 04 01:30:11.848882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.811521 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" event={"ID":"9027ba5f-d082-4126-9708-82766ac7e612","Type":"ContainerDied","Data":"890cd1373fa0f18d52cff292516d425f191acfb781fc1e93cc5887fa57a043d9"} Nov 04 01:30:11.848882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.828924 5051 scope.go:122] "RemoveContainer" containerID="890cd1373fa0f18d52cff292516d425f191acfb781fc1e93cc5887fa57a043d9" Nov 04 01:30:11.852306 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:11.852109 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d_odh-ai-gateway-infra(f066d9df-2697-461e-92db-66abc15c1788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" podUID="f066d9df-2697-461e-92db-66abc15c1788" Nov 04 01:30:11.861169 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:11.852567 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j_odh-ai-gateway-infra(9027ba5f-d082-4126-9708-82766ac7e612)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" podUID="9027ba5f-d082-4126-9708-82766ac7e612" Nov 04 01:30:11.930032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.926956 5051 generic.go:356] "Generic (PLEG): container finished" podID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerID="08650e245d31f207c73e69eccd6052c841b3125c6438157ba51772e3b1645da0" exitCode=60 Nov 04 01:30:11.930032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.927149 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" event={"ID":"b0fd0185-4925-4a4b-9c7c-67678d2da4b8","Type":"ContainerDied","Data":"08650e245d31f207c73e69eccd6052c841b3125c6438157ba51772e3b1645da0"} Nov 04 01:30:11.970887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:11.940645 5051 scope.go:122] "RemoveContainer" containerID="08650e245d31f207c73e69eccd6052c841b3125c6438157ba51772e3b1645da0" Nov 04 01:30:11.971797 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:11.941353 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827_odh-ai-gateway-infra(b0fd0185-4925-4a4b-9c7c-67678d2da4b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" Nov 04 01:30:12.300883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.161503 5051 generic.go:356] "Generic (PLEG): container finished" podID="000eb04e-a38b-424e-a668-feae478b052f" containerID="7379c08180c348a83c1f0bd5b80416e6b487ab74c142463717306e1a49b2ea9a" exitCode=60 Nov 04 01:30:12.300883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.172385 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" event={"ID":"000eb04e-a38b-424e-a668-feae478b052f","Type":"ContainerDied","Data":"7379c08180c348a83c1f0bd5b80416e6b487ab74c142463717306e1a49b2ea9a"} Nov 04 01:30:12.300883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.204672 5051 scope.go:122] "RemoveContainer" containerID="7379c08180c348a83c1f0bd5b80416e6b487ab74c142463717306e1a49b2ea9a" Nov 04 01:30:12.300883 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:12.205435 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957_odh-ai-gateway-infra(000eb04e-a38b-424e-a668-feae478b052f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" podUID="000eb04e-a38b-424e-a668-feae478b052f" Nov 04 01:30:12.426536 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.426496 5051 scope.go:122] "RemoveContainer" containerID="bdad473437a8b3ff53222fa757c05eea5c51ef09ed755941c977b72f82871acc" Nov 04 01:30:12.463175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.459611 5051 scope.go:122] "RemoveContainer" containerID="e86ed5fb74795c1fedb71f99dc87ce27eb6a2a9697a9b3c1262a0a1770116ec5" Nov 04 01:30:12.463175 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:12.460110 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj_odh-ai-gateway-infra(f4245e28-a7cf-4df5-a41f-73a0da0f78fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" Nov 04 01:30:12.643232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.591690 5051 generic.go:356] "Generic (PLEG): container finished" podID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerID="5425c58dd85e63173a3b224cd3c98c5b70e9deffdb824bde134b249d4ad55135" exitCode=60 Nov 04 01:30:12.643232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.591819 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" event={"ID":"b5f7a55e-0d37-4aeb-9888-32c89c718adf","Type":"ContainerDied","Data":"5425c58dd85e63173a3b224cd3c98c5b70e9deffdb824bde134b249d4ad55135"} Nov 04 01:30:12.643232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.592393 5051 scope.go:122] "RemoveContainer" containerID="5425c58dd85e63173a3b224cd3c98c5b70e9deffdb824bde134b249d4ad55135" Nov 04 01:30:12.643232 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:12.592878 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv_odh-ai-gateway-infra(b5f7a55e-0d37-4aeb-9888-32c89c718adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" Nov 04 01:30:12.703180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.668816 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:12.703180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.668876 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:12.907777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.907680 5051 generic.go:356] "Generic (PLEG): container finished" podID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerID="a4689bd60863305d88d5b6b9e41d68756da8de1ec8444b63e98757f321f93b12" exitCode=60 Nov 04 01:30:12.908074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.908045 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" event={"ID":"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f","Type":"ContainerDied","Data":"a4689bd60863305d88d5b6b9e41d68756da8de1ec8444b63e98757f321f93b12"} Nov 04 01:30:12.908827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.908803 5051 scope.go:122] "RemoveContainer" containerID="a4689bd60863305d88d5b6b9e41d68756da8de1ec8444b63e98757f321f93b12" Nov 04 01:30:12.909522 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:12.909490 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s_odh-ai-gateway-infra(e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" Nov 04 01:30:12.914889 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.914838 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:12.915824 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.915801 5051 generic.go:356] "Generic (PLEG): container finished" podID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerID="307a0c57a226046bd87ff8cb51fdd037b789d50fd757364cd008a4b581f83e0f" exitCode=60 Nov 04 01:30:12.916263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.915842 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" event={"ID":"4f36bcd3-2829-4dad-a069-1cb09f599e0c","Type":"ContainerDied","Data":"307a0c57a226046bd87ff8cb51fdd037b789d50fd757364cd008a4b581f83e0f"} Nov 04 01:30:12.916911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.916873 5051 scope.go:122] "RemoveContainer" containerID="307a0c57a226046bd87ff8cb51fdd037b789d50fd757364cd008a4b581f83e0f" Nov 04 01:30:12.917486 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:12.917456 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn_odh-ai-gateway-infra(4f36bcd3-2829-4dad-a069-1cb09f599e0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" Nov 04 01:30:12.919404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.919381 5051 scope.go:122] "RemoveContainer" containerID="48f084feb07de17bf84962c99f39b16d246763b62bdf152c960ebecf6274edb5" Nov 04 01:30:12.933067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.932406 5051 generic.go:356] "Generic (PLEG): container finished" podID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerID="648c71627331c8b37e2cff6f60d962244f7c4f6a79f793515bd027f8c58dde6f" exitCode=60 Nov 04 01:30:12.933067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.932758 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" event={"ID":"1d81258f-8ed3-4153-9d49-41656cb054fe","Type":"ContainerDied","Data":"648c71627331c8b37e2cff6f60d962244f7c4f6a79f793515bd027f8c58dde6f"} Nov 04 01:30:12.933325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.933305 5051 scope.go:122] "RemoveContainer" containerID="648c71627331c8b37e2cff6f60d962244f7c4f6a79f793515bd027f8c58dde6f" Nov 04 01:30:12.934787 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:12.933718 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895930-dp6m9_odh-ai-gateway-infra(1d81258f-8ed3-4153-9d49-41656cb054fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" Nov 04 01:30:12.939297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.939264 5051 generic.go:356] "Generic (PLEG): container finished" podID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerID="8964c60261c46f45d23ee193838e706620057e13f20a4fcf8d5801c0d6fa99e6" exitCode=60 Nov 04 01:30:12.939434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.939314 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" event={"ID":"4134be8d-cd30-414f-b35c-0ce688b2184f","Type":"ContainerDied","Data":"8964c60261c46f45d23ee193838e706620057e13f20a4fcf8d5801c0d6fa99e6"} Nov 04 01:30:12.940161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:12.940108 5051 scope.go:122] "RemoveContainer" containerID="8964c60261c46f45d23ee193838e706620057e13f20a4fcf8d5801c0d6fa99e6" Nov 04 01:30:12.944132 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:12.944079 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8_odh-ai-gateway-infra(4134be8d-cd30-414f-b35c-0ce688b2184f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" Nov 04 01:30:13.021394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.021360 5051 scope.go:122] "RemoveContainer" containerID="1fb5fed23c66ac5aba0f1439c50d0441cc25a9e291b83c501b0cbc31e14e4e0c" Nov 04 01:30:13.043673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.043648 5051 scope.go:122] "RemoveContainer" containerID="cda3d27eb5f7399d6157c267b82e467e128a2826111d80ae141f986e3076051a" Nov 04 01:30:13.105361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.104645 5051 scope.go:122] "RemoveContainer" containerID="b4d8afdc04a7c87e8c4197fa47428ed8a8803a6338f76df0fac5b42e97be7049" Nov 04 01:30:13.127808 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.127781 5051 scope.go:122] "RemoveContainer" containerID="a02b746271ed4bf37ffdc9c11919f7d9cab12343e3a54b3f3d9ffcf336606f20" Nov 04 01:30:13.146562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.146538 5051 scope.go:122] "RemoveContainer" containerID="0fa0f2168f9e1df0cc74eda1c35bcc6ce017b696d71da67bafd776295029df23" Nov 04 01:30:13.217780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.217756 5051 scope.go:122] "RemoveContainer" containerID="6ec6d26f027f259379edd0633bf6c2305ba434becc61bbd42cc52d93dff47e01" Nov 04 01:30:13.579587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.579553 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 04 01:30:13.584165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.579859 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 04 01:30:13.969654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:13.969624 5051 scope.go:122] "RemoveContainer" containerID="08650e245d31f207c73e69eccd6052c841b3125c6438157ba51772e3b1645da0" Nov 04 01:30:13.970752 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:13.970211 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827_odh-ai-gateway-infra(b0fd0185-4925-4a4b-9c7c-67678d2da4b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" Nov 04 01:30:14.584930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:14.584888 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:30:14.585369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:14.585338 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:30:14.586223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:14.586201 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:30:15.075144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:15.048534 5051 generic.go:356] "Generic (PLEG): container finished" podID="e1c36e46-07ec-47df-a65c-d35233518e01" containerID="45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a" exitCode=0 Nov 04 01:30:15.075144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:15.050024 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" event={"ID":"e1c36e46-07ec-47df-a65c-d35233518e01","Type":"ContainerDied","Data":"45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a"} Nov 04 01:30:15.075144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:15.073735 5051 scope.go:122] "RemoveContainer" containerID="45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a" Nov 04 01:30:15.080269 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:15.077733 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:15.080269 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:15.077918 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:15.080269 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:15.077968 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:16.133603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:16.133565 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" event={"ID":"e1c36e46-07ec-47df-a65c-d35233518e01","Type":"ContainerStarted","Data":"cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85"} Nov 04 01:30:16.923615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:16.920445 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:30:16.931658 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:16.923939 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:30:16.931658 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:16.924020 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:30:17.940144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:17.934879 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:30:17.940144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:17.934955 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:30:18.434938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:18.434900 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:30:18.925147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:18.921996 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:30:18.925147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:18.922062 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:30:19.939147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:19.938183 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:30:19.939147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:19.938249 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:30:21.086154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:21.086085 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:30:21.086699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:21.086185 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:30:21.200146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:21.195687 5051 scope.go:122] "RemoveContainer" containerID="08650e245d31f207c73e69eccd6052c841b3125c6438157ba51772e3b1645da0" Nov 04 01:30:21.200146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:21.196286 5051 scope.go:122] "RemoveContainer" containerID="5425c58dd85e63173a3b224cd3c98c5b70e9deffdb824bde134b249d4ad55135" Nov 04 01:30:22.235206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:22.155630 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:30:22.235206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:22.156213 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:30:22.397683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:22.245061 5051 scope.go:122] "RemoveContainer" containerID="307a0c57a226046bd87ff8cb51fdd037b789d50fd757364cd008a4b581f83e0f" Nov 04 01:30:22.624465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:22.573712 5051 scope.go:122] "RemoveContainer" containerID="e86ed5fb74795c1fedb71f99dc87ce27eb6a2a9697a9b3c1262a0a1770116ec5" Nov 04 01:30:22.624465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:22.614465 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:30:22.847142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:22.843271 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:30:22.847142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:22.843397 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:30:23.150505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.145877 5051 scope.go:122] "RemoveContainer" containerID="890cd1373fa0f18d52cff292516d425f191acfb781fc1e93cc5887fa57a043d9" Nov 04 01:30:23.150505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.146977 5051 scope.go:122] "RemoveContainer" containerID="648c71627331c8b37e2cff6f60d962244f7c4f6a79f793515bd027f8c58dde6f" Nov 04 01:30:23.160282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.150971 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:30:23.181146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.172815 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:23.210494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.207343 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:30:23.210494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.207413 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:30:23.565752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.283919 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:30:23.565752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.535623 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:23.565752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.538189 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:30:23.565752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.545843 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:30:23.593894 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.590465 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:23.605102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.590775 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:23.605410 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.593721 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:30:23.741288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.741257 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:30:23.744012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.743992 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:30:23.744812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.744794 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf"] Nov 04 01:30:23.744972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.744960 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx"] Nov 04 01:30:23.745072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.745064 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:30:23.747929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.746247 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:30:23.747929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.742725 5051 scope.go:122] "RemoveContainer" containerID="7379c08180c348a83c1f0bd5b80416e6b487ab74c142463717306e1a49b2ea9a" Nov 04 01:30:23.747929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.747473 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" containerID="cri-o://90d0a5039e4a9f5eba4f4c8719046e15417cdc2a1c9d8247ef616321af104fa2" gracePeriod=30 Nov 04 01:30:23.747929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.747675 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" containerID="cri-o://e23f651ae1fc6fb819d0667cfd4f7fbbaf7196fed0c7e2d7c7ab9fb3d0926e8e" gracePeriod=30 Nov 04 01:30:23.747929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.747755 5051 scope.go:122] "RemoveContainer" containerID="8964c60261c46f45d23ee193838e706620057e13f20a4fcf8d5801c0d6fa99e6" Nov 04 01:30:23.748636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.748199 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb"] Nov 04 01:30:23.748636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.748346 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" containerID="cri-o://e1ea6229a2c7fb80f2eb2d426eb6f44ffefd2f6d290bf91d435e6ec3a0e24170" gracePeriod=30 Nov 04 01:30:23.793275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.793037 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz"] Nov 04 01:30:23.794235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.794202 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" containerID="cri-o://58e152501bdacba54c6f9683cf4a6572a98cd4a2a4fedf9b345559c786a74387" gracePeriod=30 Nov 04 01:30:23.927163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.922675 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 04 01:30:23.927163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:23.922744 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 04 01:30:24.181499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.181469 5051 scope.go:122] "RemoveContainer" containerID="a4689bd60863305d88d5b6b9e41d68756da8de1ec8444b63e98757f321f93b12" Nov 04 01:30:24.841200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.828476 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" event={"ID":"1d81258f-8ed3-4153-9d49-41656cb054fe","Type":"ContainerStarted","Data":"179e071cc7401fbc058150f361da88426f6d789a6af08ea09bd1900783ccdae0"} Nov 04 01:30:24.841200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.835390 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" event={"ID":"4134be8d-cd30-414f-b35c-0ce688b2184f","Type":"ContainerStarted","Data":"b5cd9a79adcf4b618d4b90a445877d6486ebf41d1b5a1b07bbbeaa9b1f8164b4"} Nov 04 01:30:24.841200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.839199 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" event={"ID":"9027ba5f-d082-4126-9708-82766ac7e612","Type":"ContainerStarted","Data":"6c3aeb4a03996641e541a92e2a843c648769f65fd3ba5542d03153aa93bad494"} Nov 04 01:30:24.842532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.842506 5051 generic.go:356] "Generic (PLEG): container finished" podID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerID="2eb0fee84db4b2bbb778047c6a6b9d2748ce1504c6f71bebce3fad94ffeb6ba0" exitCode=60 Nov 04 01:30:24.842688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.842600 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" event={"ID":"b0fd0185-4925-4a4b-9c7c-67678d2da4b8","Type":"ContainerDied","Data":"2eb0fee84db4b2bbb778047c6a6b9d2748ce1504c6f71bebce3fad94ffeb6ba0"} Nov 04 01:30:24.842688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.842633 5051 scope.go:122] "RemoveContainer" containerID="08650e245d31f207c73e69eccd6052c841b3125c6438157ba51772e3b1645da0" Nov 04 01:30:24.842982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.842967 5051 scope.go:122] "RemoveContainer" containerID="2eb0fee84db4b2bbb778047c6a6b9d2748ce1504c6f71bebce3fad94ffeb6ba0" Nov 04 01:30:24.843280 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:24.843260 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827_odh-ai-gateway-infra(b0fd0185-4925-4a4b-9c7c-67678d2da4b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" Nov 04 01:30:24.846181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.846153 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" event={"ID":"000eb04e-a38b-424e-a668-feae478b052f","Type":"ContainerStarted","Data":"26e99aaedb204b51bc4b905b5ef204079bc9b2f85daf068b8e3e8f0aaf32b67b"} Nov 04 01:30:24.847950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.847923 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" event={"ID":"f4245e28-a7cf-4df5-a41f-73a0da0f78fa","Type":"ContainerStarted","Data":"ec27c657661acdd1d360e904f92345973eecd100795189db6f975245150092da"} Nov 04 01:30:24.862165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.862098 5051 generic.go:356] "Generic (PLEG): container finished" podID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerID="2ee3ce492a8c59926c7ba785158714b58e3f467263f1ed33d3f23d6ce2efb03c" exitCode=60 Nov 04 01:30:24.862370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.862207 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" event={"ID":"b5f7a55e-0d37-4aeb-9888-32c89c718adf","Type":"ContainerDied","Data":"2ee3ce492a8c59926c7ba785158714b58e3f467263f1ed33d3f23d6ce2efb03c"} Nov 04 01:30:24.863257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.863228 5051 scope.go:122] "RemoveContainer" containerID="2ee3ce492a8c59926c7ba785158714b58e3f467263f1ed33d3f23d6ce2efb03c" Nov 04 01:30:24.863610 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:24.863589 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv_odh-ai-gateway-infra(b5f7a55e-0d37-4aeb-9888-32c89c718adf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" Nov 04 01:30:24.865263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.865235 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" event={"ID":"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f","Type":"ContainerStarted","Data":"d82a5e4a4f13ab2ccfd668fbb90067f712a536795eb46b0f3c6d2d5bd33cf567"} Nov 04 01:30:24.872258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.871204 5051 scope.go:122] "RemoveContainer" containerID="5425c58dd85e63173a3b224cd3c98c5b70e9deffdb824bde134b249d4ad55135" Nov 04 01:30:24.873596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.873573 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" event={"ID":"4f36bcd3-2829-4dad-a069-1cb09f599e0c","Type":"ContainerStarted","Data":"6cccb8d2da8cf6233091353ac8f6a80f484ebb222ffe8f2c238f332e55a118d3"} Nov 04 01:30:24.929346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.929260 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:30:24.930852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:24.930825 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:30:25.701038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.700932 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s"] Nov 04 01:30:25.874475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.874439 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9"] Nov 04 01:30:25.896380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.895042 5051 generic.go:356] "Generic (PLEG): container finished" podID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerID="d82a5e4a4f13ab2ccfd668fbb90067f712a536795eb46b0f3c6d2d5bd33cf567" exitCode=60 Nov 04 01:30:25.896380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.895197 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" event={"ID":"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f","Type":"ContainerDied","Data":"d82a5e4a4f13ab2ccfd668fbb90067f712a536795eb46b0f3c6d2d5bd33cf567"} Nov 04 01:30:25.896380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.895262 5051 scope.go:122] "RemoveContainer" containerID="a4689bd60863305d88d5b6b9e41d68756da8de1ec8444b63e98757f321f93b12" Nov 04 01:30:25.903567 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.903539 5051 generic.go:356] "Generic (PLEG): container finished" podID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerID="6cccb8d2da8cf6233091353ac8f6a80f484ebb222ffe8f2c238f332e55a118d3" exitCode=60 Nov 04 01:30:25.903787 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.903750 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" event={"ID":"4f36bcd3-2829-4dad-a069-1cb09f599e0c","Type":"ContainerDied","Data":"6cccb8d2da8cf6233091353ac8f6a80f484ebb222ffe8f2c238f332e55a118d3"} Nov 04 01:30:25.904365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.904341 5051 scope.go:122] "RemoveContainer" containerID="6cccb8d2da8cf6233091353ac8f6a80f484ebb222ffe8f2c238f332e55a118d3" Nov 04 01:30:25.904957 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:25.904919 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn_odh-ai-gateway-infra(4f36bcd3-2829-4dad-a069-1cb09f599e0c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" Nov 04 01:30:25.907348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.907321 5051 generic.go:356] "Generic (PLEG): container finished" podID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerID="179e071cc7401fbc058150f361da88426f6d789a6af08ea09bd1900783ccdae0" exitCode=60 Nov 04 01:30:25.908670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.908530 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" event={"ID":"1d81258f-8ed3-4153-9d49-41656cb054fe","Type":"ContainerDied","Data":"179e071cc7401fbc058150f361da88426f6d789a6af08ea09bd1900783ccdae0"} Nov 04 01:30:25.912273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.912191 5051 generic.go:356] "Generic (PLEG): container finished" podID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerID="b5cd9a79adcf4b618d4b90a445877d6486ebf41d1b5a1b07bbbeaa9b1f8164b4" exitCode=60 Nov 04 01:30:25.912445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.912405 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" event={"ID":"4134be8d-cd30-414f-b35c-0ce688b2184f","Type":"ContainerDied","Data":"b5cd9a79adcf4b618d4b90a445877d6486ebf41d1b5a1b07bbbeaa9b1f8164b4"} Nov 04 01:30:25.912899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.912833 5051 scope.go:122] "RemoveContainer" containerID="b5cd9a79adcf4b618d4b90a445877d6486ebf41d1b5a1b07bbbeaa9b1f8164b4" Nov 04 01:30:25.913315 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:25.913273 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8_odh-ai-gateway-infra(4134be8d-cd30-414f-b35c-0ce688b2184f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" Nov 04 01:30:25.916245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.916214 5051 generic.go:356] "Generic (PLEG): container finished" podID="9027ba5f-d082-4126-9708-82766ac7e612" containerID="6c3aeb4a03996641e541a92e2a843c648769f65fd3ba5542d03153aa93bad494" exitCode=60 Nov 04 01:30:25.916530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.916462 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" event={"ID":"9027ba5f-d082-4126-9708-82766ac7e612","Type":"ContainerDied","Data":"6c3aeb4a03996641e541a92e2a843c648769f65fd3ba5542d03153aa93bad494"} Nov 04 01:30:25.919710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.919691 5051 scope.go:122] "RemoveContainer" containerID="6c3aeb4a03996641e541a92e2a843c648769f65fd3ba5542d03153aa93bad494" Nov 04 01:30:25.924439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.924418 5051 scope.go:122] "RemoveContainer" containerID="307a0c57a226046bd87ff8cb51fdd037b789d50fd757364cd008a4b581f83e0f" Nov 04 01:30:25.925752 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:25.925697 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j_odh-ai-gateway-infra(9027ba5f-d082-4126-9708-82766ac7e612)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" podUID="9027ba5f-d082-4126-9708-82766ac7e612" Nov 04 01:30:25.926654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.926631 5051 generic.go:356] "Generic (PLEG): container finished" podID="000eb04e-a38b-424e-a668-feae478b052f" containerID="26e99aaedb204b51bc4b905b5ef204079bc9b2f85daf068b8e3e8f0aaf32b67b" exitCode=60 Nov 04 01:30:25.926759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.926739 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" event={"ID":"000eb04e-a38b-424e-a668-feae478b052f","Type":"ContainerDied","Data":"26e99aaedb204b51bc4b905b5ef204079bc9b2f85daf068b8e3e8f0aaf32b67b"} Nov 04 01:30:25.927331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.927304 5051 scope.go:122] "RemoveContainer" containerID="26e99aaedb204b51bc4b905b5ef204079bc9b2f85daf068b8e3e8f0aaf32b67b" Nov 04 01:30:25.927746 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:25.927693 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957_odh-ai-gateway-infra(000eb04e-a38b-424e-a668-feae478b052f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" podUID="000eb04e-a38b-424e-a668-feae478b052f" Nov 04 01:30:25.931816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.931256 5051 generic.go:356] "Generic (PLEG): container finished" podID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerID="ec27c657661acdd1d360e904f92345973eecd100795189db6f975245150092da" exitCode=60 Nov 04 01:30:25.931816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.931362 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" event={"ID":"f4245e28-a7cf-4df5-a41f-73a0da0f78fa","Type":"ContainerDied","Data":"ec27c657661acdd1d360e904f92345973eecd100795189db6f975245150092da"} Nov 04 01:30:25.932002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.931913 5051 scope.go:122] "RemoveContainer" containerID="ec27c657661acdd1d360e904f92345973eecd100795189db6f975245150092da" Nov 04 01:30:25.932480 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:25.932451 5051 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj_odh-ai-gateway-infra(f4245e28-a7cf-4df5-a41f-73a0da0f78fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" Nov 04 01:30:25.940102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:25.940060 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj"] Nov 04 01:30:26.029376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.029341 5051 scope.go:122] "RemoveContainer" containerID="648c71627331c8b37e2cff6f60d962244f7c4f6a79f793515bd027f8c58dde6f" Nov 04 01:30:26.055957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.052627 5051 scope.go:122] "RemoveContainer" containerID="8964c60261c46f45d23ee193838e706620057e13f20a4fcf8d5801c0d6fa99e6" Nov 04 01:30:26.121749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.121723 5051 scope.go:122] "RemoveContainer" containerID="890cd1373fa0f18d52cff292516d425f191acfb781fc1e93cc5887fa57a043d9" Nov 04 01:30:26.151896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.150616 5051 scope.go:122] "RemoveContainer" containerID="7379c08180c348a83c1f0bd5b80416e6b487ab74c142463717306e1a49b2ea9a" Nov 04 01:30:26.163108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.163080 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827"] Nov 04 01:30:26.185475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.185363 5051 scope.go:122] "RemoveContainer" containerID="e86ed5fb74795c1fedb71f99dc87ce27eb6a2a9697a9b3c1262a0a1770116ec5" Nov 04 01:30:26.191055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.190987 5051 scope.go:122] "RemoveContainer" containerID="d874347e421aade3534a9cdc4bd52c78cca20ec815987eeaa1a0cfd0987710a7" Nov 04 01:30:26.267763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.267733 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j"] Nov 04 01:30:26.281088 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.280999 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8"] Nov 04 01:30:26.325963 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.325939 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957"] Nov 04 01:30:26.326748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.326397 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:26.374886 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.374313 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv"] Nov 04 01:30:26.412081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.410852 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-maas-api-service-ca\") pod \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\" (UID: \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\") " Nov 04 01:30:26.412081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.410945 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-kube-api-access-26dqk\" (UniqueName: \"kubernetes.io/projected/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-kube-api-access-26dqk\") pod \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\" (UID: \"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f\") " Nov 04 01:30:26.412081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.411551 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-maas-api-service-ca" pod "e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" (UID: "e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:26.413669 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.413091 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn"] Nov 04 01:30:26.414452 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.414400 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:26.456284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.456251 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-kube-api-access-26dqk" pod "e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" (UID: "e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f"). InnerVolumeSpecName "kube-api-access-26dqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:26.552146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.540852 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26dqk\" (UniqueName: \"kubernetes.io/projected/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f-kube-api-access-26dqk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:26.607609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.607574 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:26.652192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.645114 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d81258f-8ed3-4153-9d49-41656cb054fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81258f-8ed3-4153-9d49-41656cb054fe-maas-api-service-ca\") pod \"1d81258f-8ed3-4153-9d49-41656cb054fe\" (UID: \"1d81258f-8ed3-4153-9d49-41656cb054fe\") " Nov 04 01:30:26.652192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.648537 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d81258f-8ed3-4153-9d49-41656cb054fe-kube-api-access-f5ngs\" (UniqueName: \"kubernetes.io/projected/1d81258f-8ed3-4153-9d49-41656cb054fe-kube-api-access-f5ngs\") pod \"1d81258f-8ed3-4153-9d49-41656cb054fe\" (UID: \"1d81258f-8ed3-4153-9d49-41656cb054fe\") " Nov 04 01:30:26.656772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.656298 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d81258f-8ed3-4153-9d49-41656cb054fe-kube-api-access-f5ngs" pod "1d81258f-8ed3-4153-9d49-41656cb054fe" (UID: "1d81258f-8ed3-4153-9d49-41656cb054fe"). InnerVolumeSpecName "kube-api-access-f5ngs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:26.656772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.656643 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d81258f-8ed3-4153-9d49-41656cb054fe-maas-api-service-ca" pod "1d81258f-8ed3-4153-9d49-41656cb054fe" (UID: "1d81258f-8ed3-4153-9d49-41656cb054fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:26.732259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.732100 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:26.765291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.756845 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d81258f-8ed3-4153-9d49-41656cb054fe-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:26.765291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.756873 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5ngs\" (UniqueName: \"kubernetes.io/projected/1d81258f-8ed3-4153-9d49-41656cb054fe-kube-api-access-f5ngs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:26.858564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.858525 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-maas-api-service-ca\") pod \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\" (UID: \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\") " Nov 04 01:30:26.858759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.858736 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-kube-api-access-l89lj\" (UniqueName: \"kubernetes.io/projected/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-kube-api-access-l89lj\") pod \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\" (UID: \"b0fd0185-4925-4a4b-9c7c-67678d2da4b8\") " Nov 04 01:30:26.860477 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.860446 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-maas-api-service-ca" pod "b0fd0185-4925-4a4b-9c7c-67678d2da4b8" (UID: "b0fd0185-4925-4a4b-9c7c-67678d2da4b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:26.865500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.863821 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-kube-api-access-l89lj" pod "b0fd0185-4925-4a4b-9c7c-67678d2da4b8" (UID: "b0fd0185-4925-4a4b-9c7c-67678d2da4b8"). InnerVolumeSpecName "kube-api-access-l89lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:26.964289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.964212 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:26.964801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.964773 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l89lj\" (UniqueName: \"kubernetes.io/projected/b0fd0185-4925-4a4b-9c7c-67678d2da4b8-kube-api-access-l89lj\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:26.968993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.967403 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:26.968993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.968830 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" event={"ID":"f066d9df-2697-461e-92db-66abc15c1788","Type":"ContainerStarted","Data":"37e41083c4fe4d98e7aa982b752bf1d16e7bbda64397f8f33c3d9975f18ab7ff"} Nov 04 01:30:26.972978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.972958 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" Nov 04 01:30:26.973089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.973008 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827" event={"ID":"b0fd0185-4925-4a4b-9c7c-67678d2da4b8","Type":"ContainerDied","Data":"a5bb10b6d2b96f8603edc6f21c0f4f8a1c2505f2883d1c866e93d3bf4bc12062"} Nov 04 01:30:26.973089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.973070 5051 scope.go:122] "RemoveContainer" containerID="2eb0fee84db4b2bbb778047c6a6b9d2748ce1504c6f71bebce3fad94ffeb6ba0" Nov 04 01:30:26.981853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.981759 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" Nov 04 01:30:26.981853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.981794 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv" event={"ID":"b5f7a55e-0d37-4aeb-9888-32c89c718adf","Type":"ContainerDied","Data":"f18ceb228a83647621cf29412918637ce8543faeb5774eab7f26653db1b78990"} Nov 04 01:30:26.989218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.989115 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" event={"ID":"e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f","Type":"ContainerDied","Data":"cb0c8c6de1026934b8a7e320609576d456d041559e36d8714c587a1a39b3cadb"} Nov 04 01:30:26.990045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.989382 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s" Nov 04 01:30:27.001546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:26.999764 5051 scope.go:122] "RemoveContainer" containerID="2ee3ce492a8c59926c7ba785158714b58e3f467263f1ed33d3f23d6ce2efb03c" Nov 04 01:30:27.028531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.028475 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827"] Nov 04 01:30:27.030596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.030566 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" Nov 04 01:30:27.043231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.032228 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9" event={"ID":"1d81258f-8ed3-4153-9d49-41656cb054fe","Type":"ContainerDied","Data":"d5f5815ab0df2f8501cc5f6df51466b58d6dcb6e4025dc487902db66d3426c48"} Nov 04 01:30:27.048070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.044815 5051 scope.go:122] "RemoveContainer" containerID="d82a5e4a4f13ab2ccfd668fbb90067f712a536795eb46b0f3c6d2d5bd33cf567" Nov 04 01:30:27.049600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.048837 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ca55c1-29895930-m7827"] Nov 04 01:30:27.085509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.085461 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f7a55e-0d37-4aeb-9888-32c89c718adf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f7a55e-0d37-4aeb-9888-32c89c718adf-maas-api-service-ca\") pod \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\" (UID: \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\") " Nov 04 01:30:27.086432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.086398 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f7a55e-0d37-4aeb-9888-32c89c718adf-kube-api-access-6cp75\" (UniqueName: \"kubernetes.io/projected/b5f7a55e-0d37-4aeb-9888-32c89c718adf-kube-api-access-6cp75\") pod \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\" (UID: \"b5f7a55e-0d37-4aeb-9888-32c89c718adf\") " Nov 04 01:30:27.091999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.090767 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f7a55e-0d37-4aeb-9888-32c89c718adf-maas-api-service-ca" pod "b5f7a55e-0d37-4aeb-9888-32c89c718adf" (UID: "b5f7a55e-0d37-4aeb-9888-32c89c718adf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.095946 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.095923 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f7a55e-0d37-4aeb-9888-32c89c718adf-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.099735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.099611 5051 scope.go:122] "RemoveContainer" containerID="179e071cc7401fbc058150f361da88426f6d789a6af08ea09bd1900783ccdae0" Nov 04 01:30:27.145308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.144894 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f7a55e-0d37-4aeb-9888-32c89c718adf-kube-api-access-6cp75" pod "b5f7a55e-0d37-4aeb-9888-32c89c718adf" (UID: "b5f7a55e-0d37-4aeb-9888-32c89c718adf"). InnerVolumeSpecName "kube-api-access-6cp75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.206743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.202425 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cp75\" (UniqueName: \"kubernetes.io/projected/b5f7a55e-0d37-4aeb-9888-32c89c718adf-kube-api-access-6cp75\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.276908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.276869 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" path="/var/lib/kubelet/pods/b0fd0185-4925-4a4b-9c7c-67678d2da4b8/volumes" Nov 04 01:30:27.326367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.326108 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s"] Nov 04 01:30:27.385158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.373626 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:27.402700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.399869 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d"] Nov 04 01:30:27.491408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.490442 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-9c8bf5-29895930-t4d7s"] Nov 04 01:30:27.546951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.546834 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9"] Nov 04 01:30:27.556590 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.556530 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895930-dp6m9"] Nov 04 01:30:27.574915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.574889 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv"] Nov 04 01:30:27.581864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.581830 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-df6da4c0-29895930-sxzqv"] Nov 04 01:30:27.586524 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.586501 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:27.653299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.651650 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9027ba5f-d082-4126-9708-82766ac7e612-kube-api-access-rn844\" (UniqueName: \"kubernetes.io/projected/9027ba5f-d082-4126-9708-82766ac7e612-kube-api-access-rn844\") pod \"9027ba5f-d082-4126-9708-82766ac7e612\" (UID: \"9027ba5f-d082-4126-9708-82766ac7e612\") " Nov 04 01:30:27.653299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.651774 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4134be8d-cd30-414f-b35c-0ce688b2184f-kube-api-access-hw5bx\" (UniqueName: \"kubernetes.io/projected/4134be8d-cd30-414f-b35c-0ce688b2184f-kube-api-access-hw5bx\") pod \"4134be8d-cd30-414f-b35c-0ce688b2184f\" (UID: \"4134be8d-cd30-414f-b35c-0ce688b2184f\") " Nov 04 01:30:27.653299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.651833 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4134be8d-cd30-414f-b35c-0ce688b2184f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134be8d-cd30-414f-b35c-0ce688b2184f-maas-api-service-ca\") pod \"4134be8d-cd30-414f-b35c-0ce688b2184f\" (UID: \"4134be8d-cd30-414f-b35c-0ce688b2184f\") " Nov 04 01:30:27.653299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.652745 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9027ba5f-d082-4126-9708-82766ac7e612-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9027ba5f-d082-4126-9708-82766ac7e612-maas-api-service-ca\") pod \"9027ba5f-d082-4126-9708-82766ac7e612\" (UID: \"9027ba5f-d082-4126-9708-82766ac7e612\") " Nov 04 01:30:27.653529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.653374 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4134be8d-cd30-414f-b35c-0ce688b2184f-maas-api-service-ca" pod "4134be8d-cd30-414f-b35c-0ce688b2184f" (UID: "4134be8d-cd30-414f-b35c-0ce688b2184f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.654335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.654310 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9027ba5f-d082-4126-9708-82766ac7e612-maas-api-service-ca" pod "9027ba5f-d082-4126-9708-82766ac7e612" (UID: "9027ba5f-d082-4126-9708-82766ac7e612"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.655262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.655241 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134be8d-cd30-414f-b35c-0ce688b2184f-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.655380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.655368 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9027ba5f-d082-4126-9708-82766ac7e612-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.677141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.677099 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4134be8d-cd30-414f-b35c-0ce688b2184f-kube-api-access-hw5bx" pod "4134be8d-cd30-414f-b35c-0ce688b2184f" (UID: "4134be8d-cd30-414f-b35c-0ce688b2184f"). InnerVolumeSpecName "kube-api-access-hw5bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.722060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.713946 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9027ba5f-d082-4126-9708-82766ac7e612-kube-api-access-rn844" pod "9027ba5f-d082-4126-9708-82766ac7e612" (UID: "9027ba5f-d082-4126-9708-82766ac7e612"). InnerVolumeSpecName "kube-api-access-rn844". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.795176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.790053 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn844\" (UniqueName: \"kubernetes.io/projected/9027ba5f-d082-4126-9708-82766ac7e612-kube-api-access-rn844\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.795176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.790094 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5bx\" (UniqueName: \"kubernetes.io/projected/4134be8d-cd30-414f-b35c-0ce688b2184f-kube-api-access-hw5bx\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.824191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.824165 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:27.830401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.830381 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:27.836754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.836726 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:27.894072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.893489 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f36bcd3-2829-4dad-a069-1cb09f599e0c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36bcd3-2829-4dad-a069-1cb09f599e0c-maas-api-service-ca\") pod \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\" (UID: \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\") " Nov 04 01:30:27.894072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.893712 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-maas-api-service-ca\") pod \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\" (UID: \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\") " Nov 04 01:30:27.894072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.893788 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000eb04e-a38b-424e-a668-feae478b052f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000eb04e-a38b-424e-a668-feae478b052f-maas-api-service-ca\") pod \"000eb04e-a38b-424e-a668-feae478b052f\" (UID: \"000eb04e-a38b-424e-a668-feae478b052f\") " Nov 04 01:30:27.894072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.893914 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-kube-api-access-n2gfw\" (UniqueName: \"kubernetes.io/projected/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-kube-api-access-n2gfw\") pod \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\" (UID: \"f4245e28-a7cf-4df5-a41f-73a0da0f78fa\") " Nov 04 01:30:27.894072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.894008 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f36bcd3-2829-4dad-a069-1cb09f599e0c-kube-api-access-mcsd7\" (UniqueName: \"kubernetes.io/projected/4f36bcd3-2829-4dad-a069-1cb09f599e0c-kube-api-access-mcsd7\") pod \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\" (UID: \"4f36bcd3-2829-4dad-a069-1cb09f599e0c\") " Nov 04 01:30:27.894533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.894147 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000eb04e-a38b-424e-a668-feae478b052f-kube-api-access-wb7m6\" (UniqueName: \"kubernetes.io/projected/000eb04e-a38b-424e-a668-feae478b052f-kube-api-access-wb7m6\") pod \"000eb04e-a38b-424e-a668-feae478b052f\" (UID: \"000eb04e-a38b-424e-a668-feae478b052f\") " Nov 04 01:30:27.896689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.896451 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-maas-api-service-ca" pod "f4245e28-a7cf-4df5-a41f-73a0da0f78fa" (UID: "f4245e28-a7cf-4df5-a41f-73a0da0f78fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.897163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.897094 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000eb04e-a38b-424e-a668-feae478b052f-maas-api-service-ca" pod "000eb04e-a38b-424e-a668-feae478b052f" (UID: "000eb04e-a38b-424e-a668-feae478b052f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.898444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.898346 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f36bcd3-2829-4dad-a069-1cb09f599e0c-maas-api-service-ca" pod "4f36bcd3-2829-4dad-a069-1cb09f599e0c" (UID: "4f36bcd3-2829-4dad-a069-1cb09f599e0c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:27.902570 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.900451 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f36bcd3-2829-4dad-a069-1cb09f599e0c-kube-api-access-mcsd7" pod "4f36bcd3-2829-4dad-a069-1cb09f599e0c" (UID: "4f36bcd3-2829-4dad-a069-1cb09f599e0c"). InnerVolumeSpecName "kube-api-access-mcsd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.902570 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.902181 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-kube-api-access-n2gfw" pod "f4245e28-a7cf-4df5-a41f-73a0da0f78fa" (UID: "f4245e28-a7cf-4df5-a41f-73a0da0f78fa"). InnerVolumeSpecName "kube-api-access-n2gfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.906139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.905866 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000eb04e-a38b-424e-a668-feae478b052f-kube-api-access-wb7m6" pod "000eb04e-a38b-424e-a668-feae478b052f" (UID: "000eb04e-a38b-424e-a668-feae478b052f"). InnerVolumeSpecName "kube-api-access-wb7m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:27.963927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.963887 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:27.964099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.963956 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:27.997226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.997191 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.997226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.997227 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/000eb04e-a38b-424e-a668-feae478b052f-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.997644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.997243 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2gfw\" (UniqueName: \"kubernetes.io/projected/f4245e28-a7cf-4df5-a41f-73a0da0f78fa-kube-api-access-n2gfw\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.997644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.997259 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcsd7\" (UniqueName: \"kubernetes.io/projected/4f36bcd3-2829-4dad-a069-1cb09f599e0c-kube-api-access-mcsd7\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.997644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.997273 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7m6\" (UniqueName: \"kubernetes.io/projected/000eb04e-a38b-424e-a668-feae478b052f-kube-api-access-wb7m6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:27.997644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:27.997289 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36bcd3-2829-4dad-a069-1cb09f599e0c-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.053640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.053589 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" Nov 04 01:30:28.053859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.053589 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8" event={"ID":"4134be8d-cd30-414f-b35c-0ce688b2184f","Type":"ContainerDied","Data":"85e4a8cd841759b5f6780fc3ea9312a83385ec1d23b319690bfcf99c98f7921a"} Nov 04 01:30:28.054036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.054019 5051 scope.go:122] "RemoveContainer" containerID="b5cd9a79adcf4b618d4b90a445877d6486ebf41d1b5a1b07bbbeaa9b1f8164b4" Nov 04 01:30:28.058169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.057547 5051 generic.go:356] "Generic (PLEG): container finished" podID="f066d9df-2697-461e-92db-66abc15c1788" containerID="37e41083c4fe4d98e7aa982b752bf1d16e7bbda64397f8f33c3d9975f18ab7ff" exitCode=60 Nov 04 01:30:28.058169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.057654 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" event={"ID":"f066d9df-2697-461e-92db-66abc15c1788","Type":"ContainerDied","Data":"37e41083c4fe4d98e7aa982b752bf1d16e7bbda64397f8f33c3d9975f18ab7ff"} Nov 04 01:30:28.060073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.060039 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" event={"ID":"9027ba5f-d082-4126-9708-82766ac7e612","Type":"ContainerDied","Data":"60983a393a58f75eaec872ce4e4acb5ac21a72af8fb16ab5414d1d2fae845ab5"} Nov 04 01:30:28.060227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.060033 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j" Nov 04 01:30:28.065384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.065340 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" Nov 04 01:30:28.067783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.065255 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957" event={"ID":"000eb04e-a38b-424e-a668-feae478b052f","Type":"ContainerDied","Data":"de7afdf43d080f2d786f54dcd04b1d4da3e74a82bc3da93296dc1c588257faab"} Nov 04 01:30:28.072335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.072214 5051 scope.go:122] "RemoveContainer" containerID="d874347e421aade3534a9cdc4bd52c78cca20ec815987eeaa1a0cfd0987710a7" Nov 04 01:30:28.073045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.072723 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" event={"ID":"f4245e28-a7cf-4df5-a41f-73a0da0f78fa","Type":"ContainerDied","Data":"ee476971b30aaa7779a4cfba91a105a08d2fa7eb91b2b471976e22a435e9e0f1"} Nov 04 01:30:28.073045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.072795 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj" Nov 04 01:30:28.078704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.078676 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" event={"ID":"4f36bcd3-2829-4dad-a069-1cb09f599e0c","Type":"ContainerDied","Data":"dbcf4ea1cc6a81ecf1f87fe3388b2b326219638c75aa93d21227f4b98a260f12"} Nov 04 01:30:28.078846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.078827 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn" Nov 04 01:30:28.124796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.122399 5051 scope.go:122] "RemoveContainer" containerID="6c3aeb4a03996641e541a92e2a843c648769f65fd3ba5542d03153aa93bad494" Nov 04 01:30:28.126556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.126502 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8"] Nov 04 01:30:28.131509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.131488 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-016d3d-29895930-lg5l8"] Nov 04 01:30:28.171285 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.171253 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj"] Nov 04 01:30:28.206149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.206109 5051 scope.go:122] "RemoveContainer" containerID="26e99aaedb204b51bc4b905b5ef204079bc9b2f85daf068b8e3e8f0aaf32b67b" Nov 04 01:30:28.215811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.215781 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-ef6cc9-29895930-q4ktj"] Nov 04 01:30:28.234764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.234738 5051 scope.go:122] "RemoveContainer" containerID="ec27c657661acdd1d360e904f92345973eecd100795189db6f975245150092da" Nov 04 01:30:28.296306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.296278 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j"] Nov 04 01:30:28.313378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.313319 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-5f572a-29895930-tw98j"] Nov 04 01:30:28.313798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.313701 5051 scope.go:122] "RemoveContainer" containerID="6cccb8d2da8cf6233091353ac8f6a80f484ebb222ffe8f2c238f332e55a118d3" Nov 04 01:30:28.324796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.324770 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn"] Nov 04 01:30:28.376251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.376216 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-af6909-29895930-fkhxn"] Nov 04 01:30:28.548667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.544349 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:28.580670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.580586 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957"] Nov 04 01:30:28.594929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.594900 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d7e45d-29895930-b2957"] Nov 04 01:30:28.646303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.646272 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs"] Nov 04 01:30:28.646511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.646489 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" containerID="cri-o://7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c" gracePeriod=30 Nov 04 01:30:28.647820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.647633 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f066d9df-2697-461e-92db-66abc15c1788-kube-api-access-q7bfn\" (UniqueName: \"kubernetes.io/projected/f066d9df-2697-461e-92db-66abc15c1788-kube-api-access-q7bfn\") pod \"f066d9df-2697-461e-92db-66abc15c1788\" (UID: \"f066d9df-2697-461e-92db-66abc15c1788\") " Nov 04 01:30:28.647820 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.647720 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f066d9df-2697-461e-92db-66abc15c1788-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f066d9df-2697-461e-92db-66abc15c1788-maas-api-service-ca\") pod \"f066d9df-2697-461e-92db-66abc15c1788\" (UID: \"f066d9df-2697-461e-92db-66abc15c1788\") " Nov 04 01:30:28.649032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.649003 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f066d9df-2697-461e-92db-66abc15c1788-maas-api-service-ca" pod "f066d9df-2697-461e-92db-66abc15c1788" (UID: "f066d9df-2697-461e-92db-66abc15c1788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:28.658639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.658610 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4"] Nov 04 01:30:28.658890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.658859 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" containerID="cri-o://2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5" gracePeriod=30 Nov 04 01:30:28.661989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.661963 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f066d9df-2697-461e-92db-66abc15c1788-kube-api-access-q7bfn" pod "f066d9df-2697-461e-92db-66abc15c1788" (UID: "f066d9df-2697-461e-92db-66abc15c1788"). InnerVolumeSpecName "kube-api-access-q7bfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:28.751204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.751115 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7bfn\" (UniqueName: \"kubernetes.io/projected/f066d9df-2697-461e-92db-66abc15c1788-kube-api-access-q7bfn\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.751373 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.751210 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f066d9df-2697-461e-92db-66abc15c1788-maas-api-service-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:28.761860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.761805 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7"] Nov 04 01:30:28.762797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.762761 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" containerID="cri-o://13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c" gracePeriod=30 Nov 04 01:30:28.783700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.781987 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v"] Nov 04 01:30:28.783700 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:28.782278 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" containerID="cri-o://33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a" gracePeriod=30 Nov 04 01:30:29.422713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.422687 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000eb04e-a38b-424e-a668-feae478b052f" path="/var/lib/kubelet/pods/000eb04e-a38b-424e-a668-feae478b052f/volumes" Nov 04 01:30:29.423807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.423786 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" path="/var/lib/kubelet/pods/1d81258f-8ed3-4153-9d49-41656cb054fe/volumes" Nov 04 01:30:29.424501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.424482 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" path="/var/lib/kubelet/pods/4134be8d-cd30-414f-b35c-0ce688b2184f/volumes" Nov 04 01:30:29.424845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.424819 5051 generic.go:356] "Generic (PLEG): container finished" podID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerID="e23f651ae1fc6fb819d0667cfd4f7fbbaf7196fed0c7e2d7c7ab9fb3d0926e8e" exitCode=0 Nov 04 01:30:29.427977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.425647 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" path="/var/lib/kubelet/pods/4f36bcd3-2829-4dad-a069-1cb09f599e0c/volumes" Nov 04 01:30:29.427977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.426267 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9027ba5f-d082-4126-9708-82766ac7e612" path="/var/lib/kubelet/pods/9027ba5f-d082-4126-9708-82766ac7e612/volumes" Nov 04 01:30:29.427977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.426875 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" path="/var/lib/kubelet/pods/b5f7a55e-0d37-4aeb-9888-32c89c718adf/volumes" Nov 04 01:30:29.427977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.427931 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" path="/var/lib/kubelet/pods/e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f/volumes" Nov 04 01:30:29.428987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.428967 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" path="/var/lib/kubelet/pods/f4245e28-a7cf-4df5-a41f-73a0da0f78fa/volumes" Nov 04 01:30:29.429654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.429633 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" event={"ID":"924177ce-ffbd-40a4-bb8d-0157f301d4f3","Type":"ContainerDied","Data":"e23f651ae1fc6fb819d0667cfd4f7fbbaf7196fed0c7e2d7c7ab9fb3d0926e8e"} Nov 04 01:30:29.437326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.437056 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" Nov 04 01:30:29.437955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.437422 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d" event={"ID":"f066d9df-2697-461e-92db-66abc15c1788","Type":"ContainerDied","Data":"a3f1d64f0f9b9e756a30fa13c84fcbb1a2bf717032f50e9c4e80d2dde0136823"} Nov 04 01:30:29.437955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.437475 5051 scope.go:122] "RemoveContainer" containerID="37e41083c4fe4d98e7aa982b752bf1d16e7bbda64397f8f33c3d9975f18ab7ff" Nov 04 01:30:29.442762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.442606 5051 generic.go:356] "Generic (PLEG): container finished" podID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerID="90d0a5039e4a9f5eba4f4c8719046e15417cdc2a1c9d8247ef616321af104fa2" exitCode=0 Nov 04 01:30:29.442762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.442705 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" event={"ID":"9ab06cee-b976-406b-a5b6-1073613a5bad","Type":"ContainerDied","Data":"90d0a5039e4a9f5eba4f4c8719046e15417cdc2a1c9d8247ef616321af104fa2"} Nov 04 01:30:29.448866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.448845 5051 generic.go:356] "Generic (PLEG): container finished" podID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerID="e1ea6229a2c7fb80f2eb2d426eb6f44ffefd2f6d290bf91d435e6ec3a0e24170" exitCode=0 Nov 04 01:30:29.449068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.448920 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" event={"ID":"a3170ff2-51dd-4c54-a3ea-8494c86040c9","Type":"ContainerDied","Data":"e1ea6229a2c7fb80f2eb2d426eb6f44ffefd2f6d290bf91d435e6ec3a0e24170"} Nov 04 01:30:29.459641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.459613 5051 generic.go:356] "Generic (PLEG): container finished" podID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerID="58e152501bdacba54c6f9683cf4a6572a98cd4a2a4fedf9b345559c786a74387" exitCode=0 Nov 04 01:30:29.459902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.459878 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" event={"ID":"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf","Type":"ContainerDied","Data":"58e152501bdacba54c6f9683cf4a6572a98cd4a2a4fedf9b345559c786a74387"} Nov 04 01:30:29.505688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.505658 5051 scope.go:122] "RemoveContainer" containerID="f27228c0ef1e0692373eebc2f18ef46d8f2b15d15eed76bcb07e5a5259018091" Nov 04 01:30:29.510070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.510043 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:30:29.537087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.536157 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d"] Nov 04 01:30:29.543918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.543885 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3888c434-29895930-mzt9d"] Nov 04 01:30:29.549175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.549008 5051 scope.go:122] "RemoveContainer" containerID="8c216ce625cc4c6293d29178a4759983155cef886d19dad479ea44318d758ef9" Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.626949 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-credential-socket\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.627074 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-socket\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.627366 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-podinfo\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.627470 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-envoy\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.627625 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-certs\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.627727 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-token\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-token\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.627781 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istiod-ca-cert\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.627829 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-kube-api-access-mw7vf\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-kube-api-access-mw7vf\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.635783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.627991 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-data\") pod \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\" (UID: \"a3170ff2-51dd-4c54-a3ea-8494c86040c9\") " Nov 04 01:30:29.638622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.638252 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-data" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:29.660192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.648913 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-certs" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:29.660932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.660475 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-socket" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:29.662478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.662428 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-credential-socket" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:29.664005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.663978 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istiod-ca-cert" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:29.948906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.878304 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:29.948906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.878335 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:29.948906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.878348 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:29.948906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.878360 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:29.948906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.878372 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:29.958936 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.958894 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-envoy" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.006859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.975892 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:30.006859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.975961 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:30.006859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.976214 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:30.006859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:29.976238 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:30.006859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.003517 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-token" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:30.009106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.009062 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.027790 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.027703 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-podinfo" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:30.097957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.097545 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:30:30.117671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.114273 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.117671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.114309 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.118680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.118431 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-kube-api-access-mw7vf" pod "a3170ff2-51dd-4c54-a3ea-8494c86040c9" (UID: "a3170ff2-51dd-4c54-a3ea-8494c86040c9"). InnerVolumeSpecName "kube-api-access-mw7vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.220749 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-credential-socket\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.220883 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-data\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.221019 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-certs\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.221069 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-socket\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.221101 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-envoy\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.221319 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-kube-api-access-cwv9m\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-kube-api-access-cwv9m\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.221538 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-token\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-token\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.221607 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab06cee-b976-406b-a5b6-1073613a5bad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab06cee-b976-406b-a5b6-1073613a5bad-istiod-ca-cert\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.221658 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-podinfo\") pod \"9ab06cee-b976-406b-a5b6-1073613a5bad\" (UID: \"9ab06cee-b976-406b-a5b6-1073613a5bad\") " Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.222700 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7vf\" (UniqueName: \"kubernetes.io/projected/a3170ff2-51dd-4c54-a3ea-8494c86040c9-kube-api-access-mw7vf\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.226071 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-certs" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.226667 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-data" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.226867 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-credential-socket" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.227496 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab06cee-b976-406b-a5b6-1073613a5bad-istiod-ca-cert" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.229216 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-socket" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.235598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.235325 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-kube-api-access-cwv9m" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "kube-api-access-cwv9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:30.251191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.249798 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-podinfo" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:30.251191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.249902 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-envoy" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.251191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.250023 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-token" pod "9ab06cee-b976-406b-a5b6-1073613a5bad" (UID: "9ab06cee-b976-406b-a5b6-1073613a5bad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:30.271248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.270803 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:30:30.278887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.277415 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:30:30.323954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.323904 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-token\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-token\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.324164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.323973 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-certs\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.324164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324002 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-data\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.324164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324050 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-token\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-token\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.324164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324082 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-podinfo\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.324381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324305 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istiod-ca-cert\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.324381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324342 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-socket\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.324487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324401 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-podinfo\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.324487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324418 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-data" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.324487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324446 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-socket\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.324645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324528 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-credential-socket\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.324645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324559 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-envoy\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.324645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324593 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-kube-api-access-tdbfz\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-kube-api-access-tdbfz\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.324812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324649 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istiod-ca-cert\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.324812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324416 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-certs" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.324812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324754 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-socket" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.324812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324643 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-socket" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.325015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324862 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-credential-socket" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.325015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324747 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istiod-ca-cert" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:30.325015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.324877 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-credential-socket\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.325015 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.325000 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-kube-api-access-sw8pt\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-kube-api-access-sw8pt\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.325243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.325056 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-envoy\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.325243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.325083 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-credential-socket" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.325243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.325152 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istiod-ca-cert" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:30.325243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.325174 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-certs\") pod \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\" (UID: \"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf\") " Nov 04 01:30:30.325243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.325231 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-data\") pod \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\" (UID: \"924177ce-ffbd-40a4-bb8d-0157f301d4f3\") " Nov 04 01:30:30.326304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326284 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326307 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326322 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326337 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ab06cee-b976-406b-a5b6-1073613a5bad-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326353 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326406 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326424 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326436 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326449 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326463 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326481 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326496 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326512 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326527 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ab06cee-b976-406b-a5b6-1073613a5bad-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326542 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwv9m\" (UniqueName: \"kubernetes.io/projected/9ab06cee-b976-406b-a5b6-1073613a5bad-kube-api-access-cwv9m\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326558 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.326608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326574 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.327429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326859 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-podinfo" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:30.327429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.326902 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-data" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.327429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.327209 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-certs" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.327686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.327661 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-token" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:30.327776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.327686 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-podinfo" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:30.328283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.328258 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-token" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:30.328608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.328583 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-kube-api-access-tdbfz" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "kube-api-access-tdbfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:30.328761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.328743 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-envoy" pod "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" (UID: "3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.328898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.328884 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-kube-api-access-sw8pt" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "kube-api-access-sw8pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:30.329101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.329089 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-envoy" pod "924177ce-ffbd-40a4-bb8d-0157f301d4f3" (UID: "924177ce-ffbd-40a4-bb8d-0157f301d4f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:30.428698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428661 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428696 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428724 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdbfz\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-kube-api-access-tdbfz\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428738 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw8pt\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-kube-api-access-sw8pt\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428752 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428765 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428778 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428791 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428804 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.429111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.428818 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924177ce-ffbd-40a4-bb8d-0157f301d4f3-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:30.472876 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.472832 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" Nov 04 01:30:30.473091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.472901 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz" event={"ID":"3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf","Type":"ContainerDied","Data":"73c3d9e2d9bb71e6865da33db5183ddfa73cc26f83b2cc318d1cd0be6f1dc7b9"} Nov 04 01:30:30.473091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.472989 5051 scope.go:122] "RemoveContainer" containerID="58e152501bdacba54c6f9683cf4a6572a98cd4a2a4fedf9b345559c786a74387" Nov 04 01:30:30.475436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.475408 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" event={"ID":"924177ce-ffbd-40a4-bb8d-0157f301d4f3","Type":"ContainerDied","Data":"c2eafbf21b152f26c83729a50077a20ee1d5d5ed7cbc1efca7e092bfab1c4c4a"} Nov 04 01:30:30.475587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.475526 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" Nov 04 01:30:30.491174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.486903 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" Nov 04 01:30:30.491174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.486994 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf" event={"ID":"9ab06cee-b976-406b-a5b6-1073613a5bad","Type":"ContainerDied","Data":"78a989093ed1adc8b6d0bd8e4e8ad7cb10a386353e618034d96790cae33b246d"} Nov 04 01:30:30.501742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.500950 5051 scope.go:122] "RemoveContainer" containerID="e23f651ae1fc6fb819d0667cfd4f7fbbaf7196fed0c7e2d7c7ab9fb3d0926e8e" Nov 04 01:30:30.502939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.502875 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" event={"ID":"a3170ff2-51dd-4c54-a3ea-8494c86040c9","Type":"ContainerDied","Data":"f06f47bebecad8023b775b66bde8b04a32dd87c0fca4975245392631b236b3c1"} Nov 04 01:30:30.503061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.502944 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb" Nov 04 01:30:30.520590 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.520565 5051 scope.go:122] "RemoveContainer" containerID="90d0a5039e4a9f5eba4f4c8719046e15417cdc2a1c9d8247ef616321af104fa2" Nov 04 01:30:30.525047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.525018 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz"] Nov 04 01:30:30.532834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.532806 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-frppz"] Nov 04 01:30:30.536009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.535989 5051 scope.go:122] "RemoveContainer" containerID="e1ea6229a2c7fb80f2eb2d426eb6f44ffefd2f6d290bf91d435e6ec3a0e24170" Nov 04 01:30:30.549517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.549453 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx"] Nov 04 01:30:30.554988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.554961 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx"] Nov 04 01:30:30.594396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.594322 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb"] Nov 04 01:30:30.599273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.599246 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ptjzb"] Nov 04 01:30:30.607362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.607334 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf"] Nov 04 01:30:30.658141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:30.658022 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-ckgtf"] Nov 04 01:30:31.104742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.104655 5051 patch_prober.go:29] interesting pod/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:30:31.104742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.104721 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ee876a-openshift-default-b6b6f4797-nlwpx" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:30:31.196415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.196381 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" path="/var/lib/kubelet/pods/3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf/volumes" Nov 04 01:30:31.197056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.197032 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" path="/var/lib/kubelet/pods/924177ce-ffbd-40a4-bb8d-0157f301d4f3/volumes" Nov 04 01:30:31.197598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.197580 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" path="/var/lib/kubelet/pods/9ab06cee-b976-406b-a5b6-1073613a5bad/volumes" Nov 04 01:30:31.198431 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.198412 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" path="/var/lib/kubelet/pods/a3170ff2-51dd-4c54-a3ea-8494c86040c9/volumes" Nov 04 01:30:31.198906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.198889 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f066d9df-2697-461e-92db-66abc15c1788" path="/var/lib/kubelet/pods/f066d9df-2697-461e-92db-66abc15c1788/volumes" Nov 04 01:30:31.919232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.919194 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6"] Nov 04 01:30:31.921073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921030 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:31.921258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921242 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:31.921371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921361 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:31.921445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921435 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:31.921538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921529 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" Nov 04 01:30:31.921686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921673 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" Nov 04 01:30:31.921763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921754 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:31.921835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921826 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:31.921920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921912 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:31.921991 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.921982 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:31.922068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922059 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:31.922156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922145 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:31.922229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922220 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:31.922308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922298 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:31.922439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922429 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" Nov 04 01:30:31.922511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922502 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" Nov 04 01:30:31.922601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922592 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:31.922669 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922660 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:31.922739 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922730 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:31.922805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922796 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:31.922906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922895 5051 cpu_manager.go:414] "containerMap: removing container" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" Nov 04 01:30:31.922980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.922970 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" Nov 04 01:30:31.923063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923054 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:31.923222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923210 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:31.923306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923297 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:31.923384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923374 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:31.923451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923442 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:31.923514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923506 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:31.923587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923578 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:31.923653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923645 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:31.923713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923704 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:31.923795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923785 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:31.923869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923861 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:31.923940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923931 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:31.924007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.923999 5051 cpu_manager.go:414] "containerMap: removing container" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924048 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924062 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924069 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924084 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924093 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924107 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924131 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924144 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924151 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924166 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924173 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924197 5051 cpu_manager.go:414] "containerMap: removing container" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924204 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924239 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924245 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924255 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924262 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924278 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924285 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924300 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924306 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924318 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924325 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924540 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924555 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924566 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924576 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924587 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924596 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924606 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924615 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924627 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924637 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924645 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf1dc55-cd11-4f3e-beb7-bfafd0e5bddf" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924656 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924664 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924673 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924685 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924696 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab06cee-b976-406b-a5b6-1073613a5bad" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924707 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924716 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924724 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924732 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3170ff2-51dd-4c54-a3ea-8494c86040c9" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924744 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924753 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924761 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924775 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="924177ce-ffbd-40a4-bb8d-0157f301d4f3" containerName="istio-proxy" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924784 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924792 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924804 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924814 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:31.925927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:31.924823 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:32.140174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.095399 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6"] Nov 04 01:30:32.140174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.095571 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.161260 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.161338 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.161414 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8qs\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-kube-api-access-cb8qs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.161469 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.161607 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.161748 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.161824 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.161962 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.169654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.162027 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.232151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.231509 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:32.232151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.231595 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:32.272344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.272300 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.273478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.273446 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.276295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.276268 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.276941 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.276919 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.283335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.283309 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.283900 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.283880 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.284066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.284050 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.284246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.284231 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8qs\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-kube-api-access-cb8qs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.284359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.284332 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-credential-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.284434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.277348 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-socket\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.284434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.278663 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-envoy\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.284434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.276751 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-data\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.284588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.283664 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-podinfo\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.284687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.284668 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.285664 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.285642 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istiod-ca-cert\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.286137 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.286067 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-certs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.310724 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.310600 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8qs\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-kube-api-access-cb8qs\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.344750 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.344716 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-token\") pod \"e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.547731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.547679 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:32.964034 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:30:32.964002 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ac22d4_8a67_457f_b0b7_fee2d9ee93ea.slice/crio-adede04078a8b42567a22398f3bf748c5010410b0992b1861e27c8b8c51385d9 WatchSource:0}: Error finding container adede04078a8b42567a22398f3bf748c5010410b0992b1861e27c8b8c51385d9: Status 404 returned error can't find the container with id adede04078a8b42567a22398f3bf748c5010410b0992b1861e27c8b8c51385d9 Nov 04 01:30:32.970685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.969844 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:32.970685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.969949 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:32.970685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.969992 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:32.987025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:32.986953 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6"] Nov 04 01:30:33.169794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:33.169305 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:30:33.169794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:33.169368 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:30:33.775027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:33.762856 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" event={"ID":"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea","Type":"ContainerStarted","Data":"e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25"} Nov 04 01:30:33.775027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:33.762929 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" event={"ID":"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea","Type":"ContainerStarted","Data":"adede04078a8b42567a22398f3bf748c5010410b0992b1861e27c8b8c51385d9"} Nov 04 01:30:33.799753 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:33.799689 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" podStartSLOduration=2.799671157 podStartE2EDuration="2.799671157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:33.794889172 +0000 UTC m=+1972.766638043" watchObservedRunningTime="2026-11-04 01:30:33.799671157 +0000 UTC m=+1972.771420028" Nov 04 01:30:33.925544 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:33.925497 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054d98d6_5005_40a5_9243_8c0414ea2dbf.slice/crio-33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:30:34.117141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.107393 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:30:34.213244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.211638 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:30:34.214965 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:34.214903 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244028 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-token\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-token\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244088 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-socket\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244150 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-kube-api-access-mb95x\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-kube-api-access-mb95x\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244185 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-podinfo\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244222 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-data\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244361 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-credential-socket\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244632 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-certs\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244761 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-envoy\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.245031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.244799 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istiod-ca-cert\") pod \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\" (UID: \"c3c7e3fb-2235-4375-9ed4-d5707052ac3b\") " Nov 04 01:30:34.248683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.247669 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istiod-ca-cert" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:34.250793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.250455 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-podinfo" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:34.299367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.276312 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-data" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.299367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.276827 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-socket" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.299367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.283096 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-certs" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.299367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.283334 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-credential-socket" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.302841 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.302784 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-envoy" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.304698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.304669 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-kube-api-access-mb95x" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "kube-api-access-mb95x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:34.315430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.315396 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-token" pod "c3c7e3fb-2235-4375-9ed4-d5707052ac3b" (UID: "c3c7e3fb-2235-4375-9ed4-d5707052ac3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348599 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-data\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348691 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf624c2c-5046-4423-8457-bf6e62aefad5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf624c2c-5046-4423-8457-bf6e62aefad5-istiod-ca-cert\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348709 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-credential-socket\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348759 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-socket\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348796 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-certs\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348828 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-token\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-token\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348877 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-podinfo\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348930 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-envoy\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.348972 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-kube-api-access-mwhfz\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-kube-api-access-mwhfz\") pod \"cf624c2c-5046-4423-8457-bf6e62aefad5\" (UID: \"cf624c2c-5046-4423-8457-bf6e62aefad5\") " Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351311 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351346 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351363 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351378 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351393 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351409 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351430 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb95x\" (UniqueName: \"kubernetes.io/projected/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-kube-api-access-mb95x\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351445 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.351458 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3c7e3fb-2235-4375-9ed4-d5707052ac3b-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.352347 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-socket" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.352545 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-certs" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.371143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.352895 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-credential-socket" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.414092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.378451 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf624c2c-5046-4423-8457-bf6e62aefad5-istiod-ca-cert" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:34.414092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.387995 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-data" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.430260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.430213 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-token" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:34.439608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.439498 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-podinfo" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:34.446819 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.446374 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-kube-api-access-mwhfz" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "kube-api-access-mwhfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.549984 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.550149 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.550206 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.565350 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.565384 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf624c2c-5046-4423-8457-bf6e62aefad5-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.565401 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.565417 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.565433 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.565448 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.565462 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.577159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.565476 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwhfz\" (UniqueName: \"kubernetes.io/projected/cf624c2c-5046-4423-8457-bf6e62aefad5-kube-api-access-mwhfz\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.627087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.626235 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-envoy" pod "cf624c2c-5046-4423-8457-bf6e62aefad5" (UID: "cf624c2c-5046-4423-8457-bf6e62aefad5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.648388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.648358 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:30:34.702083 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.701713 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:30:34.709271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.709241 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf624c2c-5046-4423-8457-bf6e62aefad5-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.808280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.808240 5051 generic.go:356] "Generic (PLEG): container finished" podID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerID="7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c" exitCode=0 Nov 04 01:30:34.808475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.808446 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" event={"ID":"c3c7e3fb-2235-4375-9ed4-d5707052ac3b","Type":"ContainerDied","Data":"7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c"} Nov 04 01:30:34.808520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.808501 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" event={"ID":"c3c7e3fb-2235-4375-9ed4-d5707052ac3b","Type":"ContainerDied","Data":"ebfc4dfeb7640b0fea70a163445b1a610f96f9f95d7fd8896e7da5caf978c66d"} Nov 04 01:30:34.808560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.808531 5051 scope.go:122] "RemoveContainer" containerID="7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c" Nov 04 01:30:34.809016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.808686 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs" Nov 04 01:30:34.811878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.811842 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-socket\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.811983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.811970 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-kube-api-access-dl5pg\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-kube-api-access-dl5pg\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.812042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812028 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-kube-api-access-rcn8q\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-kube-api-access-rcn8q\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.812110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812073 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-socket\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.812183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812115 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-envoy\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.812236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812189 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-token\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-token\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.812289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812232 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-credential-socket\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.812289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812264 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-certs\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.812394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812350 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-istio-token\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-istio-token\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.812446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812423 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-certs\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.812519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812503 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-data\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.812582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812562 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77aa33d6-5854-4669-90db-6608e0e51278-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa33d6-5854-4669-90db-6608e0e51278-istio-podinfo\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.812644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812605 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-data\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.812697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812652 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-credential-socket\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.812753 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812699 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-podinfo\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.812808 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812772 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/054d98d6-5005-40a5-9243-8c0414ea2dbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054d98d6-5005-40a5-9243-8c0414ea2dbf-istiod-ca-cert\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.812924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.812901 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77aa33d6-5854-4669-90db-6608e0e51278-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa33d6-5854-4669-90db-6608e0e51278-istiod-ca-cert\") pod \"77aa33d6-5854-4669-90db-6608e0e51278\" (UID: \"77aa33d6-5854-4669-90db-6608e0e51278\") " Nov 04 01:30:34.813094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.813077 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-envoy\") pod \"054d98d6-5005-40a5-9243-8c0414ea2dbf\" (UID: \"054d98d6-5005-40a5-9243-8c0414ea2dbf\") " Nov 04 01:30:34.818098 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.818067 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-data" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.818211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.818082 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-istio-token" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:34.819440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.818417 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-certs" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.819440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.818433 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-socket" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.819440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.818500 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-data" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.820926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.820901 5051 generic.go:356] "Generic (PLEG): container finished" podID="77aa33d6-5854-4669-90db-6608e0e51278" containerID="13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c" exitCode=0 Nov 04 01:30:34.821047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.821023 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-kube-api-access-dl5pg" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "kube-api-access-dl5pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:34.821303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.821281 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" event={"ID":"77aa33d6-5854-4669-90db-6608e0e51278","Type":"ContainerDied","Data":"13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c"} Nov 04 01:30:34.821553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.821533 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" event={"ID":"77aa33d6-5854-4669-90db-6608e0e51278","Type":"ContainerDied","Data":"be67c33f1318bd58489d39102fb86ab86438165af591eb425729616b741086be"} Nov 04 01:30:34.821689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.821663 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-credential-socket" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.821878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.821861 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7" Nov 04 01:30:34.822682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.822659 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-socket" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.824501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.823486 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-kube-api-access-rcn8q" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "kube-api-access-rcn8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:34.824501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.823747 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-credential-socket" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.824501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.823970 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-certs" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.825571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.825551 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77aa33d6-5854-4669-90db-6608e0e51278-istio-podinfo" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:34.825677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.825624 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/054d98d6-5005-40a5-9243-8c0414ea2dbf-istiod-ca-cert" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:34.826076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.826053 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77aa33d6-5854-4669-90db-6608e0e51278-istiod-ca-cert" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:34.826319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.826293 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-podinfo" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:34.829998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.828626 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-token" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:34.833875 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.830670 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-envoy" pod "054d98d6-5005-40a5-9243-8c0414ea2dbf" (UID: "054d98d6-5005-40a5-9243-8c0414ea2dbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.834395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.834364 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-envoy" pod "77aa33d6-5854-4669-90db-6608e0e51278" (UID: "77aa33d6-5854-4669-90db-6608e0e51278"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:34.839950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.839930 5051 scope.go:122] "RemoveContainer" containerID="61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d" Nov 04 01:30:34.841037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.840807 5051 generic.go:356] "Generic (PLEG): container finished" podID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerID="33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a" exitCode=0 Nov 04 01:30:34.841037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.840949 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" event={"ID":"054d98d6-5005-40a5-9243-8c0414ea2dbf","Type":"ContainerDied","Data":"33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a"} Nov 04 01:30:34.841553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.841099 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" event={"ID":"054d98d6-5005-40a5-9243-8c0414ea2dbf","Type":"ContainerDied","Data":"b78ec4be6ffade693b1927269a172fb3df52c7a11330c4a6ab51c78b3ac934f3"} Nov 04 01:30:34.842828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.842809 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v" Nov 04 01:30:34.843660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.843629 5051 generic.go:356] "Generic (PLEG): container finished" podID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerID="2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5" exitCode=0 Nov 04 01:30:34.844749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.844727 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" Nov 04 01:30:34.846207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.846185 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" event={"ID":"cf624c2c-5046-4423-8457-bf6e62aefad5","Type":"ContainerDied","Data":"2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5"} Nov 04 01:30:34.846371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.846355 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4" event={"ID":"cf624c2c-5046-4423-8457-bf6e62aefad5","Type":"ContainerDied","Data":"a7e5a947e052ae7ba8e6423674d76cca020a358efe1fb81895b1fb4763cf8b7f"} Nov 04 01:30:34.913354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.913321 5051 scope.go:122] "RemoveContainer" containerID="7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c" Nov 04 01:30:34.916506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916475 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77aa33d6-5854-4669-90db-6608e0e51278-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916511 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916528 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916544 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl5pg\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-kube-api-access-dl5pg\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916562 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcn8q\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-kube-api-access-rcn8q\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916578 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916592 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916607 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916622 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916636 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916650 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77aa33d6-5854-4669-90db-6608e0e51278-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916664 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916677 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77aa33d6-5854-4669-90db-6608e0e51278-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916691 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77aa33d6-5854-4669-90db-6608e0e51278-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916705 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916718 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/054d98d6-5005-40a5-9243-8c0414ea2dbf-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916732 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/054d98d6-5005-40a5-9243-8c0414ea2dbf-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.916881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.916745 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/054d98d6-5005-40a5-9243-8c0414ea2dbf-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:34.918481 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:34.918401 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c\": container with ID starting with 7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c not found: ID does not exist" containerID="7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c" Nov 04 01:30:34.919049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.919000 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c"} err="failed to get container status \"7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c\": rpc error: code = NotFound desc = could not find container \"7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c\": container with ID starting with 7e88ea5c030448d8e8643765524223f946cb95268b4bcd27e87e63c2538ddd0c not found: ID does not exist" Nov 04 01:30:34.919540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.919525 5051 scope.go:122] "RemoveContainer" containerID="61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d" Nov 04 01:30:34.920817 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:34.920796 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d\": container with ID starting with 61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d not found: ID does not exist" containerID="61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d" Nov 04 01:30:34.921023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.921005 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d"} err="failed to get container status \"61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d\": rpc error: code = NotFound desc = could not find container \"61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d\": container with ID starting with 61a463fdbea5001901ba578f1ad714ab3f3c8d060462b533d1cdfd7b0cb9558d not found: ID does not exist" Nov 04 01:30:34.921236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.921217 5051 scope.go:122] "RemoveContainer" containerID="13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c" Nov 04 01:30:34.925561 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.925536 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs"] Nov 04 01:30:34.933019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.932989 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-ssqjs"] Nov 04 01:30:34.941185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.941159 5051 scope.go:122] "RemoveContainer" containerID="2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3" Nov 04 01:30:34.941504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.941453 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4"] Nov 04 01:30:34.994265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:34.954913 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-zdrh4"] Nov 04 01:30:35.000153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.000090 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v"] Nov 04 01:30:35.005863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.005835 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-j7n8v"] Nov 04 01:30:35.006420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.006397 5051 scope.go:122] "RemoveContainer" containerID="13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c" Nov 04 01:30:35.006731 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:35.006704 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c\": container with ID starting with 13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c not found: ID does not exist" containerID="13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c" Nov 04 01:30:35.006817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.006738 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c"} err="failed to get container status \"13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c\": rpc error: code = NotFound desc = could not find container \"13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c\": container with ID starting with 13bbf87a6a0debd9060df2f7274e115f9cb967cdb77c88cd164c0ed095133c1c not found: ID does not exist" Nov 04 01:30:35.006817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.006758 5051 scope.go:122] "RemoveContainer" containerID="2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3" Nov 04 01:30:35.007051 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:35.007033 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3\": container with ID starting with 2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3 not found: ID does not exist" containerID="2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3" Nov 04 01:30:35.007133 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.007058 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3"} err="failed to get container status \"2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3\": rpc error: code = NotFound desc = could not find container \"2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3\": container with ID starting with 2d7f46b17c028f444dc1cf8ca845ff9d087d24050a772af1c6f99c710dce19d3 not found: ID does not exist" Nov 04 01:30:35.007133 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.007075 5051 scope.go:122] "RemoveContainer" containerID="33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a" Nov 04 01:30:35.020040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.019057 5051 scope.go:122] "RemoveContainer" containerID="08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579" Nov 04 01:30:35.032762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.032742 5051 scope.go:122] "RemoveContainer" containerID="33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a" Nov 04 01:30:35.033094 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:35.033073 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a\": container with ID starting with 33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a not found: ID does not exist" containerID="33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a" Nov 04 01:30:35.033175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.033102 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a"} err="failed to get container status \"33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a\": rpc error: code = NotFound desc = could not find container \"33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a\": container with ID starting with 33ed6b7fadac0218e8ead7e6d5384c2ba70655449a80ab8694db1b1c79dd310a not found: ID does not exist" Nov 04 01:30:35.033175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.033135 5051 scope.go:122] "RemoveContainer" containerID="08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579" Nov 04 01:30:35.033439 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:35.033422 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579\": container with ID starting with 08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579 not found: ID does not exist" containerID="08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579" Nov 04 01:30:35.033491 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.033444 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579"} err="failed to get container status \"08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579\": rpc error: code = NotFound desc = could not find container \"08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579\": container with ID starting with 08cfb4ee9e2f1f5cc247b79eaa4e8b8c7686dba2360e9d04bc4cecdc4ffe7579 not found: ID does not exist" Nov 04 01:30:35.033491 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.033458 5051 scope.go:122] "RemoveContainer" containerID="2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5" Nov 04 01:30:35.048899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.048878 5051 scope.go:122] "RemoveContainer" containerID="7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36" Nov 04 01:30:35.096519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.096497 5051 scope.go:122] "RemoveContainer" containerID="2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5" Nov 04 01:30:35.096830 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:35.096810 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5\": container with ID starting with 2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5 not found: ID does not exist" containerID="2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5" Nov 04 01:30:35.096869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.096838 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5"} err="failed to get container status \"2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5\": rpc error: code = NotFound desc = could not find container \"2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5\": container with ID starting with 2cc696e67be9245c0d27405d3426f9ef2ba2c373a4fe4550b67bdcc63c6b57a5 not found: ID does not exist" Nov 04 01:30:35.096869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.096856 5051 scope.go:122] "RemoveContainer" containerID="7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36" Nov 04 01:30:35.097218 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:35.097115 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36\": container with ID starting with 7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36 not found: ID does not exist" containerID="7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36" Nov 04 01:30:35.097218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.097156 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36"} err="failed to get container status \"7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36\": rpc error: code = NotFound desc = could not find container \"7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36\": container with ID starting with 7b277738c71e290e68a5fdc4bd15d8e4eb2e613e460724ff7e36cb748a402d36 not found: ID does not exist" Nov 04 01:30:35.202096 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.202059 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" path="/var/lib/kubelet/pods/054d98d6-5005-40a5-9243-8c0414ea2dbf/volumes" Nov 04 01:30:35.202912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.202891 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" path="/var/lib/kubelet/pods/c3c7e3fb-2235-4375-9ed4-d5707052ac3b/volumes" Nov 04 01:30:35.203592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.203574 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" path="/var/lib/kubelet/pods/cf624c2c-5046-4423-8457-bf6e62aefad5/volumes" Nov 04 01:30:35.204287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.204269 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7"] Nov 04 01:30:35.204375 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.204302 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-c89t7"] Nov 04 01:30:35.586579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.586543 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:35.586738 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:35.586606 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 04 01:30:36.581778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:36.581724 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:36.582446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:36.582388 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 04 01:30:37.220611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:37.220574 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77aa33d6-5854-4669-90db-6608e0e51278" path="/var/lib/kubelet/pods/77aa33d6-5854-4669-90db-6608e0e51278/volumes" Nov 04 01:30:37.548778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:37.548744 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:37.548941 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:37.548797 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 04 01:30:38.015974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.015930 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz"] Nov 04 01:30:38.017735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.017704 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:38.017940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.017926 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a5836d-bfd8-4ce0-b0e8-4ce6e553536f" containerName="cleanup" Nov 04 01:30:38.018041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018031 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:38.018176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018164 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:38.018313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018303 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" Nov 04 01:30:38.018395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018385 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" Nov 04 01:30:38.018466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018457 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018516 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018532 5051 cpu_manager.go:414] "containerMap: removing container" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018540 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018553 5051 cpu_manager.go:414] "containerMap: removing container" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018560 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018580 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018587 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018601 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018608 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018630 5051 cpu_manager.go:414] "containerMap: removing container" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018636 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018673 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018680 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018709 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018716 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4134be8d-cd30-414f-b35c-0ce688b2184f" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018740 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018748 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9027ba5f-d082-4126-9708-82766ac7e612" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.018951 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4245e28-a7cf-4df5-a41f-73a0da0f78fa" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.019056 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.019068 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.019082 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f36bcd3-2829-4dad-a069-1cb09f599e0c" containerName="cleanup" Nov 04 01:30:38.019180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.019091 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" Nov 04 01:30:38.022673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.022654 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" Nov 04 01:30:38.022761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.022750 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="000eb04e-a38b-424e-a668-feae478b052f" containerName="cleanup" Nov 04 01:30:38.022828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.022818 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f066d9df-2697-461e-92db-66abc15c1788" containerName="cleanup" Nov 04 01:30:38.022895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.022886 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fd0185-4925-4a4b-9c7c-67678d2da4b8" containerName="cleanup" Nov 04 01:30:38.022978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.022969 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" Nov 04 01:30:38.023046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.023038 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" Nov 04 01:30:38.023142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.023114 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d81258f-8ed3-4153-9d49-41656cb054fe" containerName="cleanup" Nov 04 01:30:38.023229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.023219 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f7a55e-0d37-4aeb-9888-32c89c718adf" containerName="cleanup" Nov 04 01:30:38.086160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.084895 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz"] Nov 04 01:30:38.086160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.085033 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.145934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.145896 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.146229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.146207 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.146433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.146415 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.146654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.146638 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.146830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.146816 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.146992 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.146978 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.147221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.147205 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvnh\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-kube-api-access-gqvnh\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.147420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.147406 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.147573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.147560 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.249900 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.249990 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.250048 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.250112 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.250260 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvnh\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-kube-api-access-gqvnh\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.250372 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.250430 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.250590 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.250616 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.264928 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-data\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.265217 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-certs\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.265477 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.266439 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istiod-ca-cert\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.266994 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-credential-socket\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.271366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.267272 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-podinfo\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.288822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.288785 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-envoy\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.289390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.289324 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-token\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.290432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.290409 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvnh\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-kube-api-access-gqvnh\") pod \"e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.410923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.410888 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:38.548804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.548654 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:38.548804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.548709 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 04 01:30:38.840064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.828890 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz"] Nov 04 01:30:38.918171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.907536 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:38.918171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.907636 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:38.918171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:38.907679 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:38.918171 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:30:38.917452 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a8fc37_9b81_44e3_ae6e_6d1c3b86b020.slice/crio-35f5725d91a7d15215bea0ba524bd85f825e90747d5e0ff35fa970c165981523 WatchSource:0}: Error finding container 35f5725d91a7d15215bea0ba524bd85f825e90747d5e0ff35fa970c165981523: Status 404 returned error can't find the container with id 35f5725d91a7d15215bea0ba524bd85f825e90747d5e0ff35fa970c165981523 Nov 04 01:30:39.023280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:39.023243 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" event={"ID":"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020","Type":"ContainerStarted","Data":"35f5725d91a7d15215bea0ba524bd85f825e90747d5e0ff35fa970c165981523"} Nov 04 01:30:39.549504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:39.549468 5051 patch_prober.go:29] interesting pod/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 04 01:30:39.549793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:39.549764 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 04 01:30:40.036813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:40.036775 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" event={"ID":"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020","Type":"ContainerStarted","Data":"dbfa6f8f004702dd7b0766d8c27b2b30e1955e146643ec7257855742e2b99d79"} Nov 04 01:30:40.083149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:40.078343 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podStartSLOduration=3.07832086 podStartE2EDuration="3.07832086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:40.075484875 +0000 UTC m=+1979.047233748" watchObservedRunningTime="2026-11-04 01:30:40.07832086 +0000 UTC m=+1979.050069749" Nov 04 01:30:40.412364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:40.412260 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:40.413395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:40.413353 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 04 01:30:40.413565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:40.413546 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 04 01:30:40.572146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:40.571649 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:40.581148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:40.574346 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:40.582397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:40.582351 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:30:41.411958 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:41.411915 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 04 01:30:41.412499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:41.411993 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 04 01:30:42.411263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:42.411223 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 04 01:30:42.411474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:42.411292 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 04 01:30:43.231211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.231176 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj"] Nov 04 01:30:43.232200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.231390 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" containerID="cri-o://a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c" gracePeriod=30 Nov 04 01:30:43.238974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.238948 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z"] Nov 04 01:30:43.239233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.239206 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" containerID="cri-o://c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487" gracePeriod=30 Nov 04 01:30:43.425232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.424594 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 04 01:30:43.425232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.424694 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 04 01:30:43.725067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.725031 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf"] Nov 04 01:30:43.725361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.725333 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" containerID="cri-o://2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9" gracePeriod=30 Nov 04 01:30:43.732903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.732866 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw"] Nov 04 01:30:43.733167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:43.733139 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" containerID="cri-o://0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4" gracePeriod=30 Nov 04 01:30:44.411863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:44.411823 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 04 01:30:44.412419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:44.411894 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 04 01:30:45.413143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:45.411573 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 04 01:30:45.413143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:45.411636 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 04 01:30:46.422004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:46.421901 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:46.422972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:46.422942 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:46.423904 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:46.423871 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:30:47.422147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.422097 5051 scope.go:122] "RemoveContainer" containerID="fd9c3003ffcb4933cf275f9a8f7078de9be0d037e1394f62d82dbd30c27950fa" Nov 04 01:30:47.443751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.443719 5051 scope.go:122] "RemoveContainer" containerID="ecccf78e3847d6de5951e2de850c918e4e412ba961132a9fe3219ead16a0c260" Nov 04 01:30:47.455893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.455875 5051 scope.go:122] "RemoveContainer" containerID="b1d51d57af98fe7f586a0076f7a9c320074c38b8673bdd249e5c9c2cbf0d2197" Nov 04 01:30:47.467903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.467884 5051 scope.go:122] "RemoveContainer" containerID="b118be58540fa403863e341d2512090d0cdcc0d0326635962eb731c9c8ed4074" Nov 04 01:30:47.483703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.483672 5051 scope.go:122] "RemoveContainer" containerID="0aff91e2e5ab7a21544f585205918ee519dbe7dc4f61879e33a9cc416f7a480a" Nov 04 01:30:47.495555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.495532 5051 scope.go:122] "RemoveContainer" containerID="44f386153a7daed5c7d30f04d1b6d36ea2f3bfd125d104af8d877df3d020d5b8" Nov 04 01:30:47.509728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.509708 5051 scope.go:122] "RemoveContainer" containerID="a48fee7b60e94ab75d08ef3539bbedccfe0f12ec0a896ed91a3a1767f89324c2" Nov 04 01:30:47.520839 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.520821 5051 scope.go:122] "RemoveContainer" containerID="982336d56e500582c764d3c3d02cff61201bb52d33948067de981ad2fc8b8265" Nov 04 01:30:47.532783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.532761 5051 scope.go:122] "RemoveContainer" containerID="c9c6fb874d1b0f3abbc14883ff7da274610d5544379e252ed8e192f6eb3747dd" Nov 04 01:30:47.545776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.545754 5051 scope.go:122] "RemoveContainer" containerID="e20a2d5a67f23ed57885c77dac6f262bc02b52569d55e520269a7f85ab50cfec" Nov 04 01:30:47.559917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.559895 5051 scope.go:122] "RemoveContainer" containerID="221a742ac7d5b3cf0356ad06bf074207d9150031f2e349b0cce2e08120ccdff0" Nov 04 01:30:47.570895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.570872 5051 scope.go:122] "RemoveContainer" containerID="d9ab30fb474faf71d233bf3cec9d7623d95a918e9c4116225fff066ea66255db" Nov 04 01:30:47.581449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.581428 5051 scope.go:122] "RemoveContainer" containerID="d0707e1e40bbf5f177ed9782d938ab50e9ff2ef55f010a9b5e3c4e8f7194ed01" Nov 04 01:30:47.594964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.594946 5051 scope.go:122] "RemoveContainer" containerID="6d2a9f5c61bf1c17243ee62457b3d9595b17f6a38b81e99e65756ddf0871f3c6" Nov 04 01:30:47.605881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:47.605859 5051 scope.go:122] "RemoveContainer" containerID="bcc72b4f33e598e90a04b40058f0c9e088f9ccd647ff60dfd2f78398bb9ae43f" Nov 04 01:30:48.564770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.564740 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.620667 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-certs\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.620715 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56eb01ae-97a9-4691-8f8b-4573f186cc87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb01ae-97a9-4691-8f8b-4573f186cc87-istiod-ca-cert\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.620806 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-data\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.620849 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-kube-api-access-9zpql\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-kube-api-access-9zpql\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.620920 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-envoy\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.621007 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-token\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-token\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.621170 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-credential-socket\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.621277 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-socket\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.621358 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-data" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.621393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.621376 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-podinfo\") pod \"56eb01ae-97a9-4691-8f8b-4573f186cc87\" (UID: \"56eb01ae-97a9-4691-8f8b-4573f186cc87\") " Nov 04 01:30:48.621974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.621608 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-certs" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.621974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.621950 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56eb01ae-97a9-4691-8f8b-4573f186cc87-istiod-ca-cert" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:48.622303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.622276 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.622427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.622308 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56eb01ae-97a9-4691-8f8b-4573f186cc87-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.622427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.622322 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.623370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.623345 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-socket" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.623479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.623423 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-credential-socket" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.625910 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.625886 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-kube-api-access-9zpql" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "kube-api-access-9zpql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:48.626043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.626021 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-podinfo" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:48.627871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.627844 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-envoy" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.628579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.628552 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-token" pod "56eb01ae-97a9-4691-8f8b-4573f186cc87" (UID: "56eb01ae-97a9-4691-8f8b-4573f186cc87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:48.632300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.632276 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:30:48.723730 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.723683 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-certs\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.723920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.723800 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fd3792-4e02-4f79-9049-8d4bd4973d48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd3792-4e02-4f79-9049-8d4bd4973d48-istiod-ca-cert\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.723920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.723863 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-socket\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.723920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.723891 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-credential-socket\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.724181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.723952 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-podinfo\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.724181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.724010 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-token\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-token\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.724181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.724029 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-data\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.724181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.724174 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-kube-api-access-7mhns\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-kube-api-access-7mhns\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.724427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.724234 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-envoy\") pod \"35fd3792-4e02-4f79-9049-8d4bd4973d48\" (UID: \"35fd3792-4e02-4f79-9049-8d4bd4973d48\") " Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.724888 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-credential-socket" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.724990 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-certs" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725157 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zpql\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-kube-api-access-9zpql\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725182 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725185 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fd3792-4e02-4f79-9049-8d4bd4973d48-istiod-ca-cert" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725197 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725237 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725254 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725271 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56eb01ae-97a9-4691-8f8b-4573f186cc87-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.725310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725287 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56eb01ae-97a9-4691-8f8b-4573f186cc87-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.726032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725440 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-socket" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.726032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.725457 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-data" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.727827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.727708 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-envoy" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:48.727827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.727810 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-token" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:48.728473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.728406 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-podinfo" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:48.733293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.729333 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-kube-api-access-7mhns" pod "35fd3792-4e02-4f79-9049-8d4bd4973d48" (UID: "35fd3792-4e02-4f79-9049-8d4bd4973d48"). InnerVolumeSpecName "kube-api-access-7mhns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:48.827600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.827186 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mhns\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-kube-api-access-7mhns\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.827600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.827225 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.827600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.827240 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.827600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.827250 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35fd3792-4e02-4f79-9049-8d4bd4973d48-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.827600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.827263 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.827600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.827276 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.827600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.827288 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:48.827600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:48.827303 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35fd3792-4e02-4f79-9049-8d4bd4973d48-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.209529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.209358 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:30:49.219235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.219206 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:30:49.248361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.248329 5051 generic.go:356] "Generic (PLEG): container finished" podID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerID="a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c" exitCode=0 Nov 04 01:30:49.248541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.248373 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" Nov 04 01:30:49.248541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.248413 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" event={"ID":"56eb01ae-97a9-4691-8f8b-4573f186cc87","Type":"ContainerDied","Data":"a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c"} Nov 04 01:30:49.248541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.248471 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj" event={"ID":"56eb01ae-97a9-4691-8f8b-4573f186cc87","Type":"ContainerDied","Data":"f0bca35da8abf06522b7d3f987a6d78d34c631984cc59b40c0fad243286289a5"} Nov 04 01:30:49.248541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.248504 5051 scope.go:122] "RemoveContainer" containerID="a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c" Nov 04 01:30:49.250724 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.250658 5051 generic.go:356] "Generic (PLEG): container finished" podID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerID="c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487" exitCode=0 Nov 04 01:30:49.250845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.250759 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" Nov 04 01:30:49.250845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.250773 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" event={"ID":"35fd3792-4e02-4f79-9049-8d4bd4973d48","Type":"ContainerDied","Data":"c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487"} Nov 04 01:30:49.250845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.250826 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z" event={"ID":"35fd3792-4e02-4f79-9049-8d4bd4973d48","Type":"ContainerDied","Data":"aaec5f62d603fdc064b6eeab51a8e6baa6046aeccb42e7d051ee1f9804102086"} Nov 04 01:30:49.257304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.257279 5051 generic.go:356] "Generic (PLEG): container finished" podID="56a3871f-4717-4043-807a-9200b309fe63" containerID="0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4" exitCode=0 Nov 04 01:30:49.257425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.257412 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" Nov 04 01:30:49.257534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.257520 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" event={"ID":"56a3871f-4717-4043-807a-9200b309fe63","Type":"ContainerDied","Data":"0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4"} Nov 04 01:30:49.257956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.257555 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw" event={"ID":"56a3871f-4717-4043-807a-9200b309fe63","Type":"ContainerDied","Data":"1a549360af3763d84b642637ebfa2e9a80437a46d5ae6485fbb3a95849d997b4"} Nov 04 01:30:49.264258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.260754 5051 generic.go:356] "Generic (PLEG): container finished" podID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerID="2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9" exitCode=0 Nov 04 01:30:49.264258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.260847 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" event={"ID":"9c27e987-5d66-42a9-8ac3-3a377697bc18","Type":"ContainerDied","Data":"2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9"} Nov 04 01:30:49.264258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.261831 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" Nov 04 01:30:49.264258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.261964 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf" event={"ID":"9c27e987-5d66-42a9-8ac3-3a377697bc18","Type":"ContainerDied","Data":"b6cd7c219d0ba51ff3c0cbcae114e5f26a3a358c0441036e99fe3f69b7ee6710"} Nov 04 01:30:49.268201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.268174 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c27e987-5d66-42a9-8ac3-3a377697bc18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c27e987-5d66-42a9-8ac3-3a377697bc18-istiod-ca-cert\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.268472 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.268452 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-socket\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.268593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.268506 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-credential-socket\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.268593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.268553 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-data\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.268714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.268634 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-certs\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.268767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.268712 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-podinfo\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.268859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.268835 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-token\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-token\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.268999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.268979 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-kube-api-access-6kvgh\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-kube-api-access-6kvgh\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.269297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.269277 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-envoy\") pod \"9c27e987-5d66-42a9-8ac3-3a377697bc18\" (UID: \"9c27e987-5d66-42a9-8ac3-3a377697bc18\") " Nov 04 01:30:49.303593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.295906 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c27e987-5d66-42a9-8ac3-3a377697bc18-istiod-ca-cert" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:49.303593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.296081 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-data" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.303593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.296706 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-credential-socket" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.303593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.297559 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-socket" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.304639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.304148 5051 scope.go:122] "RemoveContainer" containerID="a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c" Nov 04 01:30:49.304639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.304349 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-certs" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.305609 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:49.305198 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c\": container with ID starting with a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c not found: ID does not exist" containerID="a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c" Nov 04 01:30:49.305609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.305241 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c"} err="failed to get container status \"a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c\": rpc error: code = NotFound desc = could not find container \"a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c\": container with ID starting with a393d0d3826dcda2f54cf8c52acd5457c3ed36d790cf1e12ed5c6f4006e6892c not found: ID does not exist" Nov 04 01:30:49.305609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.305266 5051 scope.go:122] "RemoveContainer" containerID="c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487" Nov 04 01:30:49.309790 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.309740 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-envoy" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.309790 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.309750 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-kube-api-access-6kvgh" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "kube-api-access-6kvgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:49.309988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.309799 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-token" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:49.311982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.311249 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-podinfo" pod "9c27e987-5d66-42a9-8ac3-3a377697bc18" (UID: "9c27e987-5d66-42a9-8ac3-3a377697bc18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:49.350546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.350513 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z"] Nov 04 01:30:49.360107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.360042 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-dnb2z"] Nov 04 01:30:49.360759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.360294 5051 scope.go:122] "RemoveContainer" containerID="1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c" Nov 04 01:30:49.371548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.371525 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-envoy\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.371668 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.371630 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-data\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.371731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.371704 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a3871f-4717-4043-807a-9200b309fe63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a3871f-4717-4043-807a-9200b309fe63-istiod-ca-cert\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.371832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.371814 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-credential-socket\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.372085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.372065 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-certs\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.372231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.372209 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-credential-socket" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.372423 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.372403 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-kube-api-access-8d6zz\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-kube-api-access-8d6zz\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.373811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.372411 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-certs" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.373811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.372547 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-istio-token\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-istio-token\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.373811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.372588 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56a3871f-4717-4043-807a-9200b309fe63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a3871f-4717-4043-807a-9200b309fe63-istio-podinfo\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.373811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.372657 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-socket\") pod \"56a3871f-4717-4043-807a-9200b309fe63\" (UID: \"56a3871f-4717-4043-807a-9200b309fe63\") " Nov 04 01:30:49.373811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.373437 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a3871f-4717-4043-807a-9200b309fe63-istiod-ca-cert" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:30:49.373811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.373733 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-socket" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374051 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a3871f-4717-4043-807a-9200b309fe63-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374076 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374091 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374105 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374133 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374158 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374173 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374188 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374203 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374219 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvgh\" (UniqueName: \"kubernetes.io/projected/9c27e987-5d66-42a9-8ac3-3a377697bc18-kube-api-access-6kvgh\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374234 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c27e987-5d66-42a9-8ac3-3a377697bc18-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374248 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374265 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c27e987-5d66-42a9-8ac3-3a377697bc18-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.374954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.374520 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-data" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.375534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.375204 5051 scope.go:122] "RemoveContainer" containerID="c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487" Nov 04 01:30:49.376997 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:49.376964 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487\": container with ID starting with c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487 not found: ID does not exist" containerID="c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487" Nov 04 01:30:49.377106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.377013 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487"} err="failed to get container status \"c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487\": rpc error: code = NotFound desc = could not find container \"c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487\": container with ID starting with c5095a75a07556e248af1883951226bfe94e8017c837a109b25452ecf02ab487 not found: ID does not exist" Nov 04 01:30:49.377106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.377038 5051 scope.go:122] "RemoveContainer" containerID="1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c" Nov 04 01:30:49.377417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.377304 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-envoy" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:30:49.377529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.377509 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56a3871f-4717-4043-807a-9200b309fe63-istio-podinfo" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:30:49.377597 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:49.377532 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c\": container with ID starting with 1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c not found: ID does not exist" containerID="1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c" Nov 04 01:30:49.377597 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.377568 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c"} err="failed to get container status \"1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c\": rpc error: code = NotFound desc = could not find container \"1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c\": container with ID starting with 1772301e6a06646b863cc17ed341cd7728c0dcefd07dbf65ffbe976bf16bc74c not found: ID does not exist" Nov 04 01:30:49.377597 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.377594 5051 scope.go:122] "RemoveContainer" containerID="0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4" Nov 04 01:30:49.377720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.377535 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-kube-api-access-8d6zz" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "kube-api-access-8d6zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:49.377720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.377556 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-istio-token" pod "56a3871f-4717-4043-807a-9200b309fe63" (UID: "56a3871f-4717-4043-807a-9200b309fe63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:30:49.381376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.381353 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj"] Nov 04 01:30:49.387615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.387594 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-4n2sj"] Nov 04 01:30:49.429013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.428987 5051 scope.go:122] "RemoveContainer" containerID="0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4" Nov 04 01:30:49.429433 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:49.429400 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4\": container with ID starting with 0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4 not found: ID does not exist" containerID="0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4" Nov 04 01:30:49.429591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.429443 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4"} err="failed to get container status \"0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4\": rpc error: code = NotFound desc = could not find container \"0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4\": container with ID starting with 0982e85cf99684b7658ba4915891cc66a4d7d46746f66643378d7ba484f409f4 not found: ID does not exist" Nov 04 01:30:49.429591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.429465 5051 scope.go:122] "RemoveContainer" containerID="2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9" Nov 04 01:30:49.441365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.441340 5051 scope.go:122] "RemoveContainer" containerID="9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e" Nov 04 01:30:49.459229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.459204 5051 scope.go:122] "RemoveContainer" containerID="2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9" Nov 04 01:30:49.459560 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:49.459538 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9\": container with ID starting with 2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9 not found: ID does not exist" containerID="2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9" Nov 04 01:30:49.459642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.459570 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9"} err="failed to get container status \"2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9\": rpc error: code = NotFound desc = could not find container \"2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9\": container with ID starting with 2619eb51e2df7e6cc168b13c9f2db8eb89cbc42abe148fda0a3169e32797b2f9 not found: ID does not exist" Nov 04 01:30:49.459642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.459595 5051 scope.go:122] "RemoveContainer" containerID="9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e" Nov 04 01:30:49.459886 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:30:49.459869 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e\": container with ID starting with 9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e not found: ID does not exist" containerID="9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e" Nov 04 01:30:49.459939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.459895 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e"} err="failed to get container status \"9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e\": rpc error: code = NotFound desc = could not find container \"9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e\": container with ID starting with 9abdcc7e29e07509c3790740561effef2216db1402c15c54d71f6616f77b358e not found: ID does not exist" Nov 04 01:30:49.476462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.476434 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.476569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.476463 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a3871f-4717-4043-807a-9200b309fe63-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.476569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.476479 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d6zz\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-kube-api-access-8d6zz\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.476569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.476492 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a3871f-4717-4043-807a-9200b309fe63-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.476569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.476505 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a3871f-4717-4043-807a-9200b309fe63-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:30:49.599082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.599042 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf"] Nov 04 01:30:49.603446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.603409 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-rcjcf"] Nov 04 01:30:49.624010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.623924 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw"] Nov 04 01:30:49.632249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:49.632220 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-crhbw"] Nov 04 01:30:51.193215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.193171 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" path="/var/lib/kubelet/pods/35fd3792-4e02-4f79-9049-8d4bd4973d48/volumes" Nov 04 01:30:51.193956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.193937 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a3871f-4717-4043-807a-9200b309fe63" path="/var/lib/kubelet/pods/56a3871f-4717-4043-807a-9200b309fe63/volumes" Nov 04 01:30:51.194697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.194672 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" path="/var/lib/kubelet/pods/56eb01ae-97a9-4691-8f8b-4573f186cc87/volumes" Nov 04 01:30:51.195764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.195743 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" path="/var/lib/kubelet/pods/9c27e987-5d66-42a9-8ac3-3a377697bc18/volumes" Nov 04 01:30:51.718339 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.718304 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9"] Nov 04 01:30:51.719360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719333 5051 cpu_manager.go:414] "containerMap: removing container" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" Nov 04 01:30:51.719360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719359 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719380 5051 cpu_manager.go:414] "containerMap: removing container" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719388 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="054d98d6-5005-40a5-9243-8c0414ea2dbf" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719420 5051 cpu_manager.go:414] "containerMap: removing container" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719428 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719445 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719452 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719477 5051 cpu_manager.go:414] "containerMap: removing container" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719484 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719514 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719521 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719535 5051 cpu_manager.go:414] "containerMap: removing container" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719542 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719555 5051 cpu_manager.go:414] "containerMap: removing container" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719562 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719582 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" Nov 04 01:30:51.719588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719589 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c7e3fb-2235-4375-9ed4-d5707052ac3b" containerName="istio-proxy" Nov 04 01:30:51.720628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719726 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" Nov 04 01:30:51.720628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719747 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" Nov 04 01:30:51.720628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719756 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf624c2c-5046-4423-8457-bf6e62aefad5" containerName="istio-proxy" Nov 04 01:30:51.720628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719762 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="77aa33d6-5854-4669-90db-6608e0e51278" containerName="istio-proxy" Nov 04 01:30:51.720628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719771 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a3871f-4717-4043-807a-9200b309fe63" containerName="istio-proxy" Nov 04 01:30:51.720628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.719780 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="56eb01ae-97a9-4691-8f8b-4573f186cc87" containerName="istio-proxy" Nov 04 01:30:51.725881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.725859 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.741546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.737705 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-50dab0-openshift-default-dockercfg-nmfg2\"" Nov 04 01:30:51.748855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.748831 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9"] Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.874964 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.875238 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.875318 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3365fd0a-a069-4da8-bd68-b323fb2411ca-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.875384 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.875425 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4prln\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-kube-api-access-4prln\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.875459 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.875555 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.875625 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:51.880141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:51.875662 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.118898 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.118997 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.119054 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.119149 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.119374 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.119451 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3365fd0a-a069-4da8-bd68-b323fb2411ca-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.119513 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.119554 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4prln\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-kube-api-access-4prln\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.119584 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.120808 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.121498 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.121060 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.124998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.122869 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3365fd0a-a069-4da8-bd68-b323fb2411ca-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.124998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.124680 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.138636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.125431 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.193357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.193290 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.229948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.206717 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.229948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.208259 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.229948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.208387 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4prln\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-kube-api-access-4prln\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.354309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.352453 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:52.711057 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.711028 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9"] Nov 04 01:30:52.714880 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:30:52.714847 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3365fd0a_a069_4da8_bd68_b323fb2411ca.slice/crio-91f72e716791aa533a8e85064bcb00f46a2cb7156e592bd8d2bbdf2bdbcfced3 WatchSource:0}: Error finding container 91f72e716791aa533a8e85064bcb00f46a2cb7156e592bd8d2bbdf2bdbcfced3: Status 404 returned error can't find the container with id 91f72e716791aa533a8e85064bcb00f46a2cb7156e592bd8d2bbdf2bdbcfced3 Nov 04 01:30:52.717827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.717791 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:52.717919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.717864 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:52.717919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:52.717897 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:30:53.344515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:53.344468 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" event={"ID":"3365fd0a-a069-4da8-bd68-b323fb2411ca","Type":"ContainerStarted","Data":"7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79"} Nov 04 01:30:53.344923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:53.344572 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" event={"ID":"3365fd0a-a069-4da8-bd68-b323fb2411ca","Type":"ContainerStarted","Data":"91f72e716791aa533a8e85064bcb00f46a2cb7156e592bd8d2bbdf2bdbcfced3"} Nov 04 01:30:53.353849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:53.353808 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:53.360193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:53.360168 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:53.378691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:53.378646 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" podStartSLOduration=2.378633178 podStartE2EDuration="2.378633178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:53.369954066 +0000 UTC m=+1992.341702937" watchObservedRunningTime="2026-11-04 01:30:53.378633178 +0000 UTC m=+1992.350382118" Nov 04 01:30:54.353023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:54.352990 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:54.354038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:54.354015 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:30:57.236339 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.236303 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g"] Nov 04 01:30:57.237242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.237222 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fd3792-4e02-4f79-9049-8d4bd4973d48" containerName="istio-proxy" Nov 04 01:30:57.237302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.237246 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c27e987-5d66-42a9-8ac3-3a377697bc18" containerName="istio-proxy" Nov 04 01:30:57.241193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.241173 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:57.254174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.250189 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-50dab0\"" Nov 04 01:30:57.256197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.255400 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g"] Nov 04 01:30:57.270299 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.270266 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2"] Nov 04 01:30:57.280575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.280335 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:57.294326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.294298 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2"] Nov 04 01:30:57.314910 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.314876 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-50dab0-dockercfg-7qwzf\"" Nov 04 01:30:57.442318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.438682 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzzpr\" (UniqueName: \"kubernetes.io/projected/7527ccd3-d55b-45b8-9fe7-66553840fd76-kube-api-access-dzzpr\") pod \"payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2\" (UID: \"7527ccd3-d55b-45b8-9fe7-66553840fd76\") " pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:57.442318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.438818 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7527ccd3-d55b-45b8-9fe7-66553840fd76-plugins-config-volume\") pod \"payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2\" (UID: \"7527ccd3-d55b-45b8-9fe7-66553840fd76\") " pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:57.442318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.439093 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q276q\" (UniqueName: \"kubernetes.io/projected/4b2256e0-d640-44f4-937e-a4f3570b2211-kube-api-access-q276q\") pod \"payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g\" (UID: \"4b2256e0-d640-44f4-937e-a4f3570b2211\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:57.442318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.439160 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2256e0-d640-44f4-937e-a4f3570b2211-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g\" (UID: \"4b2256e0-d640-44f4-937e-a4f3570b2211\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:57.567148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.561840 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzzpr\" (UniqueName: \"kubernetes.io/projected/7527ccd3-d55b-45b8-9fe7-66553840fd76-kube-api-access-dzzpr\") pod \"payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2\" (UID: \"7527ccd3-d55b-45b8-9fe7-66553840fd76\") " pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:57.567148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.561994 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7527ccd3-d55b-45b8-9fe7-66553840fd76-plugins-config-volume\") pod \"payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2\" (UID: \"7527ccd3-d55b-45b8-9fe7-66553840fd76\") " pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:57.567148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.562383 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q276q\" (UniqueName: \"kubernetes.io/projected/4b2256e0-d640-44f4-937e-a4f3570b2211-kube-api-access-q276q\") pod \"payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g\" (UID: \"4b2256e0-d640-44f4-937e-a4f3570b2211\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:57.567148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.562478 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2256e0-d640-44f4-937e-a4f3570b2211-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g\" (UID: \"4b2256e0-d640-44f4-937e-a4f3570b2211\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:57.567148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.563635 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2256e0-d640-44f4-937e-a4f3570b2211-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g\" (UID: \"4b2256e0-d640-44f4-937e-a4f3570b2211\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:57.599153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.598307 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7527ccd3-d55b-45b8-9fe7-66553840fd76-plugins-config-volume\") pod \"payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2\" (UID: \"7527ccd3-d55b-45b8-9fe7-66553840fd76\") " pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:57.635145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.624747 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q276q\" (UniqueName: \"kubernetes.io/projected/4b2256e0-d640-44f4-937e-a4f3570b2211-kube-api-access-q276q\") pod \"payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g\" (UID: \"4b2256e0-d640-44f4-937e-a4f3570b2211\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:57.642215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.638450 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzzpr\" (UniqueName: \"kubernetes.io/projected/7527ccd3-d55b-45b8-9fe7-66553840fd76-kube-api-access-dzzpr\") pod \"payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2\" (UID: \"7527ccd3-d55b-45b8-9fe7-66553840fd76\") " pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:57.862842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.862740 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:57.912242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:57.912195 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:58.245738 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:30:58.245712 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2256e0_d640_44f4_937e_a4f3570b2211.slice/crio-6513d5ab0bcdcc7c041ac74e2e6708e889e671b578f45388912ae91bdf97a9a1 WatchSource:0}: Error finding container 6513d5ab0bcdcc7c041ac74e2e6708e889e671b578f45388912ae91bdf97a9a1: Status 404 returned error can't find the container with id 6513d5ab0bcdcc7c041ac74e2e6708e889e671b578f45388912ae91bdf97a9a1 Nov 04 01:30:58.269679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.269639 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g"] Nov 04 01:30:58.282890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.282855 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2"] Nov 04 01:30:58.557731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.557623 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" event={"ID":"7527ccd3-d55b-45b8-9fe7-66553840fd76","Type":"ContainerStarted","Data":"4eaa1a0909e62ae88ba61a0b467cc91e98e292c326ecc494a562b2ce53aab8dc"} Nov 04 01:30:58.557731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.557703 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:30:58.557731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.557727 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" event={"ID":"7527ccd3-d55b-45b8-9fe7-66553840fd76","Type":"ContainerStarted","Data":"3d8ec3345633d591334ae92dafb497f0c6c5f475342b02db599fb67ab83f6200"} Nov 04 01:30:58.559727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.559695 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" event={"ID":"4b2256e0-d640-44f4-937e-a4f3570b2211","Type":"ContainerStarted","Data":"8a3cf3a8859e00d6c8fdf76803e09008c24f397272f8a56a087eac237a47682f"} Nov 04 01:30:58.559883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.559756 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" event={"ID":"4b2256e0-d640-44f4-937e-a4f3570b2211","Type":"ContainerStarted","Data":"6513d5ab0bcdcc7c041ac74e2e6708e889e671b578f45388912ae91bdf97a9a1"} Nov 04 01:30:58.559883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.559797 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:30:58.589319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.589252 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" podStartSLOduration=1.589234758 podStartE2EDuration="1.589234758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:58.58420119 +0000 UTC m=+1997.555950055" watchObservedRunningTime="2026-11-04 01:30:58.589234758 +0000 UTC m=+1997.560983629" Nov 04 01:30:58.622671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:58.622613 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" podStartSLOduration=1.622598712 podStartE2EDuration="1.622598712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:30:58.619161934 +0000 UTC m=+1997.590910806" watchObservedRunningTime="2026-11-04 01:30:58.622598712 +0000 UTC m=+1997.594347636" Nov 04 01:30:59.488304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:59.488260 5051 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podf066d9df-2697-461e-92db-66abc15c1788"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podf066d9df-2697-461e-92db-66abc15c1788] : Timed out while waiting for systemd to remove kubepods-burstable-podf066d9df_2697_461e_92db_66abc15c1788.slice" Nov 04 01:30:59.488304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:30:59.488300 5051 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="f066d9df-2697-461e-92db-66abc15c1788" err="failed to delete cgroup paths for [kubepods burstable podf066d9df-2697-461e-92db-66abc15c1788] : unable to destroy cgroup paths for cgroup [kubepods burstable podf066d9df-2697-461e-92db-66abc15c1788] : Timed out while waiting for systemd to remove kubepods-burstable-podf066d9df_2697_461e_92db_66abc15c1788.slice" Nov 04 01:31:00.843579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:00.843546 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8"] Nov 04 01:31:04.226141 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:04.221692 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:31:05.844502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:05.844460 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" podUID="bbefb61b-8223-4eac-a2c0-bb9b4d2913fc" containerName="payload-processing" containerID="cri-o://90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a" gracePeriod=25 Nov 04 01:31:06.231777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.231750 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:31:06.399553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.399520 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-kube-api-access-nrfxs\" (UniqueName: \"kubernetes.io/projected/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-kube-api-access-nrfxs\") pod \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\" (UID: \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\") " Nov 04 01:31:06.399553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.399559 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-plugins-config-volume\") pod \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\" (UID: \"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc\") " Nov 04 01:31:06.400085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.400054 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-plugins-config-volume" pod "bbefb61b-8223-4eac-a2c0-bb9b4d2913fc" (UID: "bbefb61b-8223-4eac-a2c0-bb9b4d2913fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:06.400628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.400596 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:06.402040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.402010 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-kube-api-access-nrfxs" pod "bbefb61b-8223-4eac-a2c0-bb9b4d2913fc" (UID: "bbefb61b-8223-4eac-a2c0-bb9b4d2913fc"). InnerVolumeSpecName "kube-api-access-nrfxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:06.501752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.501717 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrfxs\" (UniqueName: \"kubernetes.io/projected/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc-kube-api-access-nrfxs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:06.683802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.683765 5051 generic.go:356] "Generic (PLEG): container finished" podID="bbefb61b-8223-4eac-a2c0-bb9b4d2913fc" containerID="90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a" exitCode=0 Nov 04 01:31:06.683989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.683814 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" Nov 04 01:31:06.683989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.683830 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" event={"ID":"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc","Type":"ContainerDied","Data":"90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a"} Nov 04 01:31:06.683989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.683894 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8" event={"ID":"bbefb61b-8223-4eac-a2c0-bb9b4d2913fc","Type":"ContainerDied","Data":"d17d91e6e09aa31dc0d25d60ba158e3618d81a0797a596f8da080b46e50986bf"} Nov 04 01:31:06.683989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.683925 5051 scope.go:122] "RemoveContainer" containerID="90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a" Nov 04 01:31:06.704276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.704250 5051 scope.go:122] "RemoveContainer" containerID="90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a" Nov 04 01:31:06.704655 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:06.704626 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a\": container with ID starting with 90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a not found: ID does not exist" containerID="90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a" Nov 04 01:31:06.704739 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.704666 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a"} err="failed to get container status \"90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a\": rpc error: code = NotFound desc = could not find container \"90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a\": container with ID starting with 90c8ce32334962aa63e587cd17d2ff1e650417358482750dbc5f71cf2867cd3a not found: ID does not exist" Nov 04 01:31:06.734367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.734342 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8"] Nov 04 01:31:06.762704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:06.762641 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df6da4c0-868bb67b84-twlc8"] Nov 04 01:31:07.191910 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:07.191823 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbefb61b-8223-4eac-a2c0-bb9b4d2913fc" path="/var/lib/kubelet/pods/bbefb61b-8223-4eac-a2c0-bb9b4d2913fc/volumes" Nov 04 01:31:09.495222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.495186 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7"] Nov 04 01:31:09.496386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.496363 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bbefb61b-8223-4eac-a2c0-bb9b4d2913fc" containerName="payload-processing" Nov 04 01:31:09.496472 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.496388 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbefb61b-8223-4eac-a2c0-bb9b4d2913fc" containerName="payload-processing" Nov 04 01:31:09.496606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.496593 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbefb61b-8223-4eac-a2c0-bb9b4d2913fc" containerName="payload-processing" Nov 04 01:31:09.502360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.502336 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.513303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.513253 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:31:09.513560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.513342 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:31:09.513747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.513462 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:31:09.513747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.513630 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:31:09.518076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.518055 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7"] Nov 04 01:31:09.573649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.573610 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:31:09.573821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.573731 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:31:09.662482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.662447 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.662720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.662697 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtk75\" (UniqueName: \"kubernetes.io/projected/a1ed8bbb-c795-428c-9bf0-0e0367034761-kube-api-access-rtk75\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.662922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.662901 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.764805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.764718 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.764983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.764963 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtk75\" (UniqueName: \"kubernetes.io/projected/a1ed8bbb-c795-428c-9bf0-0e0367034761-kube-api-access-rtk75\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.765316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.765284 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.766134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.766058 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.767883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.767860 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.778810 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.778785 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtk75\" (UniqueName: \"kubernetes.io/projected/a1ed8bbb-c795-428c-9bf0-0e0367034761-kube-api-access-rtk75\") pod \"maas-api-revoke-sub-44f17a39-jtfx7\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:09.820063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:09.820035 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:10.230316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:10.229386 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7"] Nov 04 01:31:10.230534 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:10.230504 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ed8bbb_c795_428c_9bf0_0e0367034761.slice/crio-4d8228343731e6a23bbea6d580e721834962380638b810105b3773475b2ffd8b WatchSource:0}: Error finding container 4d8228343731e6a23bbea6d580e721834962380638b810105b3773475b2ffd8b: Status 404 returned error can't find the container with id 4d8228343731e6a23bbea6d580e721834962380638b810105b3773475b2ffd8b Nov 04 01:31:10.733806 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:10.733772 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" event={"ID":"a1ed8bbb-c795-428c-9bf0-0e0367034761","Type":"ContainerStarted","Data":"510443b518b51739ac0837f76bb220d765cb28201cadc142f07591625dfdd368"} Nov 04 01:31:10.734208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:10.733830 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" event={"ID":"a1ed8bbb-c795-428c-9bf0-0e0367034761","Type":"ContainerStarted","Data":"4d8228343731e6a23bbea6d580e721834962380638b810105b3773475b2ffd8b"} Nov 04 01:31:10.762559 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:10.762507 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" podStartSLOduration=1.7624943260000001 podStartE2EDuration="1.762494326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:10.759328338 +0000 UTC m=+2009.731077208" watchObservedRunningTime="2026-11-04 01:31:10.762494326 +0000 UTC m=+2009.734243197" Nov 04 01:31:11.744550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:11.744513 5051 generic.go:356] "Generic (PLEG): container finished" podID="a1ed8bbb-c795-428c-9bf0-0e0367034761" containerID="510443b518b51739ac0837f76bb220d765cb28201cadc142f07591625dfdd368" exitCode=0 Nov 04 01:31:11.744954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:11.744602 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" event={"ID":"a1ed8bbb-c795-428c-9bf0-0e0367034761","Type":"ContainerDied","Data":"510443b518b51739ac0837f76bb220d765cb28201cadc142f07591625dfdd368"} Nov 04 01:31:12.949326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:12.949297 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:13.111780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.111690 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-openshift-ca\") pod \"a1ed8bbb-c795-428c-9bf0-0e0367034761\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " Nov 04 01:31:13.111934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.111883 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ed8bbb-c795-428c-9bf0-0e0367034761-kube-api-access-rtk75\" (UniqueName: \"kubernetes.io/projected/a1ed8bbb-c795-428c-9bf0-0e0367034761-kube-api-access-rtk75\") pod \"a1ed8bbb-c795-428c-9bf0-0e0367034761\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " Nov 04 01:31:13.112000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.111978 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-xks-ca\") pod \"a1ed8bbb-c795-428c-9bf0-0e0367034761\" (UID: \"a1ed8bbb-c795-428c-9bf0-0e0367034761\") " Nov 04 01:31:13.112067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.112017 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-openshift-ca" pod "a1ed8bbb-c795-428c-9bf0-0e0367034761" (UID: "a1ed8bbb-c795-428c-9bf0-0e0367034761"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:13.112902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.112872 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:13.114227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.114203 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ed8bbb-c795-428c-9bf0-0e0367034761-kube-api-access-rtk75" pod "a1ed8bbb-c795-428c-9bf0-0e0367034761" (UID: "a1ed8bbb-c795-428c-9bf0-0e0367034761"). InnerVolumeSpecName "kube-api-access-rtk75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:13.114631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.114612 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-xks-ca" pod "a1ed8bbb-c795-428c-9bf0-0e0367034761" (UID: "a1ed8bbb-c795-428c-9bf0-0e0367034761"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:31:13.215016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.214982 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtk75\" (UniqueName: \"kubernetes.io/projected/a1ed8bbb-c795-428c-9bf0-0e0367034761-kube-api-access-rtk75\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:13.215213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.215019 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ed8bbb-c795-428c-9bf0-0e0367034761-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:13.395368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.395274 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p"] Nov 04 01:31:13.396158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.396094 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a1ed8bbb-c795-428c-9bf0-0e0367034761" containerName="revoke-keys" Nov 04 01:31:13.396158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.396114 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ed8bbb-c795-428c-9bf0-0e0367034761" containerName="revoke-keys" Nov 04 01:31:13.396325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.396271 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ed8bbb-c795-428c-9bf0-0e0367034761" containerName="revoke-keys" Nov 04 01:31:13.400303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.400284 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.421085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.421064 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-385061-openshift-default-dockercfg-c4c87\"" Nov 04 01:31:13.431629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.431595 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-token\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.431748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.431723 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-podinfo\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.431791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.431755 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-data\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.431916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.431900 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-envoy\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.431959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.431928 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-credential-socket\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.431959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.431953 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istiod-ca-cert\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.432187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.432163 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-certs\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.432270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.432259 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-socket\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.432306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.432295 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrdl\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-kube-api-access-dlrdl\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.472494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.472468 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p"] Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546283 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrdl\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-kube-api-access-dlrdl\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546352 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-token\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546433 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-podinfo\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-data\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546624 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-envoy\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546648 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-credential-socket\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546683 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istiod-ca-cert\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546827 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-certs\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.547361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.546936 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-socket\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.562692 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.549153 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-data\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.606651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.606550 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-socket\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.607257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.607218 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-credential-socket\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.621648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.621560 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-certs\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.623398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.623362 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istiod-ca-cert\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.636038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.636008 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-podinfo\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.736959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.736878 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-token\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.739636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.739594 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrdl\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-kube-api-access-dlrdl\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.859354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.859309 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-envoy\") pod \"e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:13.876227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.876168 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" event={"ID":"a1ed8bbb-c795-428c-9bf0-0e0367034761","Type":"ContainerDied","Data":"4d8228343731e6a23bbea6d580e721834962380638b810105b3773475b2ffd8b"} Nov 04 01:31:13.876531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.876496 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d8228343731e6a23bbea6d580e721834962380638b810105b3773475b2ffd8b" Nov 04 01:31:13.879751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:13.879719 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7" Nov 04 01:31:14.015180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:14.015091 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:14.371253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:14.357410 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p"] Nov 04 01:31:14.450508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:14.450447 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:14.450702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:14.450630 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:14.450768 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:14.450723 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:14.891480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:14.891436 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" event={"ID":"6897ee6f-4599-48f0-b7d9-3b6484c955bc","Type":"ContainerStarted","Data":"5aa5f251ad4a72250ea7ce7c0b936a6ac23f0865ea6ab07554508a7e5114fb6e"} Nov 04 01:31:14.892457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:14.891547 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" event={"ID":"6897ee6f-4599-48f0-b7d9-3b6484c955bc","Type":"ContainerStarted","Data":"15581bc0722b53c03e5c5a2bcbb463b4e2ba756030dfe2fa4a799a6e2e26f871"} Nov 04 01:31:14.919322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:14.919272 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podStartSLOduration=1.919259488 podStartE2EDuration="1.919259488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:14.914338179 +0000 UTC m=+2013.886087061" watchObservedRunningTime="2026-11-04 01:31:14.919259488 +0000 UTC m=+2013.891008413" Nov 04 01:31:15.015515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:15.015488 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:15.021318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:15.021294 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:15.925452 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:15.925419 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:15.927923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:15.927898 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:31:17.594667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.594626 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl"] Nov 04 01:31:17.607001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.606969 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.612862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.612836 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl"] Nov 04 01:31:17.615143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.615101 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4f8dd0-openshift-default-dockercfg-xqknr\"" Nov 04 01:31:17.819078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.819037 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-socket\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.819380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.819351 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-credential-socket\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.819539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.819519 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5146a1c5-733e-4454-a89d-1071139316c1-istiod-ca-cert\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.819862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.819837 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8h9k\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-kube-api-access-f8h9k\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.820048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.820024 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-envoy\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.820532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.820506 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5146a1c5-733e-4454-a89d-1071139316c1-istio-podinfo\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.820851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.820805 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-certs\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.834381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.834352 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-istio-token\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.834650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.834622 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-data\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.942748 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-istio-token\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.942830 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-data\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.942886 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-socket\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.942914 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-credential-socket\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.942946 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5146a1c5-733e-4454-a89d-1071139316c1-istiod-ca-cert\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.943111 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h9k\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-kube-api-access-f8h9k\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.943178 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-envoy\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.943287 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5146a1c5-733e-4454-a89d-1071139316c1-istio-podinfo\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.943390 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-certs\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.943941 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-certs\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.944573 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-data\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.944855 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-socket\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:17.946190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.945093 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-credential-socket\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:18.003320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:17.983682 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5146a1c5-733e-4454-a89d-1071139316c1-istiod-ca-cert\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:18.010936 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.010064 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-envoy\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:18.014656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.014094 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5146a1c5-733e-4454-a89d-1071139316c1-istio-podinfo\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:18.023441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.023377 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-istio-token\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:18.037924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.037864 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8h9k\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-kube-api-access-f8h9k\") pod \"e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:18.233192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.233114 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:18.626163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.626130 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl"] Nov 04 01:31:18.627101 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:18.627067 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5146a1c5_733e_4454_a89d_1071139316c1.slice/crio-bda1d3c3eed4f97fc2448224b5435a012712ed1f5fafe341179ea85403b93820 WatchSource:0}: Error finding container bda1d3c3eed4f97fc2448224b5435a012712ed1f5fafe341179ea85403b93820: Status 404 returned error can't find the container with id bda1d3c3eed4f97fc2448224b5435a012712ed1f5fafe341179ea85403b93820 Nov 04 01:31:18.631064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.631022 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:18.631266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.631112 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:18.631266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.631177 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:18.701675 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.701644 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp"] Nov 04 01:31:18.709067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.709047 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:18.718830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.715943 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-385061\"" Nov 04 01:31:18.718830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.717995 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-385061-dockercfg-vfn8k\"" Nov 04 01:31:18.721025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.720925 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp"] Nov 04 01:31:18.767744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.767698 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc"] Nov 04 01:31:18.774412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.773716 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcb9033-bd09-4223-8655-7d12939824b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp\" (UID: \"3bcb9033-bd09-4223-8655-7d12939824b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:18.774412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.773769 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksk4\" (UniqueName: \"kubernetes.io/projected/3bcb9033-bd09-4223-8655-7d12939824b2-kube-api-access-sksk4\") pod \"payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp\" (UID: \"3bcb9033-bd09-4223-8655-7d12939824b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:18.801142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.780092 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:18.801142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.792623 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc"] Nov 04 01:31:18.988229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.979897 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcb9033-bd09-4223-8655-7d12939824b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp\" (UID: \"3bcb9033-bd09-4223-8655-7d12939824b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:18.988229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.979967 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksk4\" (UniqueName: \"kubernetes.io/projected/3bcb9033-bd09-4223-8655-7d12939824b2-kube-api-access-sksk4\") pod \"payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp\" (UID: \"3bcb9033-bd09-4223-8655-7d12939824b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:18.988229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.980057 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xggd5\" (UniqueName: \"kubernetes.io/projected/1530f1ae-087d-4766-aaae-02a5e6531b04-kube-api-access-xggd5\") pod \"payload-processing-e2e-api-a-385061-dd9df9788-sqqrc\" (UID: \"1530f1ae-087d-4766-aaae-02a5e6531b04\") " pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:18.988229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.980288 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f1ae-087d-4766-aaae-02a5e6531b04-plugins-config-volume\") pod \"payload-processing-e2e-api-a-385061-dd9df9788-sqqrc\" (UID: \"1530f1ae-087d-4766-aaae-02a5e6531b04\") " pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:18.988229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:18.981496 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcb9033-bd09-4223-8655-7d12939824b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp\" (UID: \"3bcb9033-bd09-4223-8655-7d12939824b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:19.109514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.061998 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksk4\" (UniqueName: \"kubernetes.io/projected/3bcb9033-bd09-4223-8655-7d12939824b2-kube-api-access-sksk4\") pod \"payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp\" (UID: \"3bcb9033-bd09-4223-8655-7d12939824b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:19.109514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.083938 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f1ae-087d-4766-aaae-02a5e6531b04-plugins-config-volume\") pod \"payload-processing-e2e-api-a-385061-dd9df9788-sqqrc\" (UID: \"1530f1ae-087d-4766-aaae-02a5e6531b04\") " pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:19.109514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.084304 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xggd5\" (UniqueName: \"kubernetes.io/projected/1530f1ae-087d-4766-aaae-02a5e6531b04-kube-api-access-xggd5\") pod \"payload-processing-e2e-api-a-385061-dd9df9788-sqqrc\" (UID: \"1530f1ae-087d-4766-aaae-02a5e6531b04\") " pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:19.109514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.102558 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f1ae-087d-4766-aaae-02a5e6531b04-plugins-config-volume\") pod \"payload-processing-e2e-api-a-385061-dd9df9788-sqqrc\" (UID: \"1530f1ae-087d-4766-aaae-02a5e6531b04\") " pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:19.373519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.373439 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:19.388643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.388566 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xggd5\" (UniqueName: \"kubernetes.io/projected/1530f1ae-087d-4766-aaae-02a5e6531b04-kube-api-access-xggd5\") pod \"payload-processing-e2e-api-a-385061-dd9df9788-sqqrc\" (UID: \"1530f1ae-087d-4766-aaae-02a5e6531b04\") " pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:19.400243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.398052 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" event={"ID":"5146a1c5-733e-4454-a89d-1071139316c1","Type":"ContainerStarted","Data":"3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3"} Nov 04 01:31:19.400243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.398111 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" event={"ID":"5146a1c5-733e-4454-a89d-1071139316c1","Type":"ContainerStarted","Data":"bda1d3c3eed4f97fc2448224b5435a012712ed1f5fafe341179ea85403b93820"} Nov 04 01:31:19.400243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.400160 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:19.444607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.436338 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podStartSLOduration=2.436313894 podStartE2EDuration="2.436313894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:19.421820026 +0000 UTC m=+2018.393568899" watchObservedRunningTime="2026-11-04 01:31:19.436313894 +0000 UTC m=+2018.408062760" Nov 04 01:31:19.883942 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:19.883910 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcb9033_bd09_4223_8655_7d12939824b2.slice/crio-33975418776fb9300f4609acc554713b24516ff9532495f6d5bc811db76d6a1d WatchSource:0}: Error finding container 33975418776fb9300f4609acc554713b24516ff9532495f6d5bc811db76d6a1d: Status 404 returned error can't find the container with id 33975418776fb9300f4609acc554713b24516ff9532495f6d5bc811db76d6a1d Nov 04 01:31:19.887904 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:19.887858 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp"] Nov 04 01:31:20.020142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.018431 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc"] Nov 04 01:31:20.020142 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:20.020007 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1530f1ae_087d_4766_aaae_02a5e6531b04.slice/crio-edf6bb32adfeb241320755b8de00c2abc6192456cd7cb07db8e6a5c546977107 WatchSource:0}: Error finding container edf6bb32adfeb241320755b8de00c2abc6192456cd7cb07db8e6a5c546977107: Status 404 returned error can't find the container with id edf6bb32adfeb241320755b8de00c2abc6192456cd7cb07db8e6a5c546977107 Nov 04 01:31:20.233741 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.233715 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:20.239245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.239216 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:20.404393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.404353 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" event={"ID":"1530f1ae-087d-4766-aaae-02a5e6531b04","Type":"ContainerStarted","Data":"aa387b00ec92e80d5fcfcc8aa69230456b4db723da5bba06188a8d7a7f5fb704"} Nov 04 01:31:20.404638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.404445 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:20.404638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.404470 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" event={"ID":"1530f1ae-087d-4766-aaae-02a5e6531b04","Type":"ContainerStarted","Data":"edf6bb32adfeb241320755b8de00c2abc6192456cd7cb07db8e6a5c546977107"} Nov 04 01:31:20.406339 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.406307 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" event={"ID":"3bcb9033-bd09-4223-8655-7d12939824b2","Type":"ContainerStarted","Data":"840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58"} Nov 04 01:31:20.406465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.406363 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" event={"ID":"3bcb9033-bd09-4223-8655-7d12939824b2","Type":"ContainerStarted","Data":"33975418776fb9300f4609acc554713b24516ff9532495f6d5bc811db76d6a1d"} Nov 04 01:31:20.406465 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.406436 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:20.406682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.406663 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:20.407401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.407379 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:31:20.424684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.424632 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" podStartSLOduration=2.424616118 podStartE2EDuration="2.424616118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:20.421572936 +0000 UTC m=+2019.393321807" watchObservedRunningTime="2026-11-04 01:31:20.424616118 +0000 UTC m=+2019.396364991" Nov 04 01:31:20.474005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:20.473939 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" podStartSLOduration=2.473918534 podStartE2EDuration="2.473918534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:20.466104036 +0000 UTC m=+2019.437852908" watchObservedRunningTime="2026-11-04 01:31:20.473918534 +0000 UTC m=+2019.445667407" Nov 04 01:31:23.388168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.388072 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7"] Nov 04 01:31:23.393362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.393339 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:23.409359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.405695 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4f8dd0\"" Nov 04 01:31:23.409359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.405798 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4f8dd0-dockercfg-v6vck\"" Nov 04 01:31:23.409737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.409715 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7"] Nov 04 01:31:23.439005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.438976 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9"] Nov 04 01:31:23.445255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.445224 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:23.454314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.454288 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9"] Nov 04 01:31:23.528148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.518478 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8z58\" (UniqueName: \"kubernetes.io/projected/af4e2767-65a2-4f53-b23a-673b34dfd0ad-kube-api-access-h8z58\") pod \"payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7\" (UID: \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:23.528148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.518799 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4e2767-65a2-4f53-b23a-673b34dfd0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7\" (UID: \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:23.633147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.629607 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4e2767-65a2-4f53-b23a-673b34dfd0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7\" (UID: \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:23.633147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.629846 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mrb\" (UniqueName: \"kubernetes.io/projected/17f25302-ecbc-4091-86d9-19234028c089-kube-api-access-m8mrb\") pod \"payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9\" (UID: \"17f25302-ecbc-4091-86d9-19234028c089\") " pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:23.633147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.629963 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8z58\" (UniqueName: \"kubernetes.io/projected/af4e2767-65a2-4f53-b23a-673b34dfd0ad-kube-api-access-h8z58\") pod \"payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7\" (UID: \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:23.633147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.630240 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f25302-ecbc-4091-86d9-19234028c089-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9\" (UID: \"17f25302-ecbc-4091-86d9-19234028c089\") " pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:23.633147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.631391 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4e2767-65a2-4f53-b23a-673b34dfd0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7\" (UID: \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:23.751144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.737307 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mrb\" (UniqueName: \"kubernetes.io/projected/17f25302-ecbc-4091-86d9-19234028c089-kube-api-access-m8mrb\") pod \"payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9\" (UID: \"17f25302-ecbc-4091-86d9-19234028c089\") " pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:23.751144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.737755 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f25302-ecbc-4091-86d9-19234028c089-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9\" (UID: \"17f25302-ecbc-4091-86d9-19234028c089\") " pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:23.751144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.738919 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f25302-ecbc-4091-86d9-19234028c089-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9\" (UID: \"17f25302-ecbc-4091-86d9-19234028c089\") " pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:23.780860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.778302 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8z58\" (UniqueName: \"kubernetes.io/projected/af4e2767-65a2-4f53-b23a-673b34dfd0ad-kube-api-access-h8z58\") pod \"payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7\" (UID: \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:23.812894 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:23.812822 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mrb\" (UniqueName: \"kubernetes.io/projected/17f25302-ecbc-4091-86d9-19234028c089-kube-api-access-m8mrb\") pod \"payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9\" (UID: \"17f25302-ecbc-4091-86d9-19234028c089\") " pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:24.010602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.010517 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:24.074140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.072654 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:24.507771 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:24.507742 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f25302_ecbc_4091_86d9_19234028c089.slice/crio-e5fab44523f5ef9e955539e73d4ac14010cb34765dd4e5c6d2504ef4218ab9d4 WatchSource:0}: Error finding container e5fab44523f5ef9e955539e73d4ac14010cb34765dd4e5c6d2504ef4218ab9d4: Status 404 returned error can't find the container with id e5fab44523f5ef9e955539e73d4ac14010cb34765dd4e5c6d2504ef4218ab9d4 Nov 04 01:31:24.509462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.509438 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9"] Nov 04 01:31:24.523030 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.523009 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7"] Nov 04 01:31:24.524075 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:24.524048 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4e2767_65a2_4f53_b23a_673b34dfd0ad.slice/crio-233310a23bd9c3f054aa5128e0d4b2aa570f39b9f99287e132792926f57f858d WatchSource:0}: Error finding container 233310a23bd9c3f054aa5128e0d4b2aa570f39b9f99287e132792926f57f858d: Status 404 returned error can't find the container with id 233310a23bd9c3f054aa5128e0d4b2aa570f39b9f99287e132792926f57f858d Nov 04 01:31:24.738311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.738274 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" event={"ID":"af4e2767-65a2-4f53-b23a-673b34dfd0ad","Type":"ContainerStarted","Data":"e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada"} Nov 04 01:31:24.738529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.738366 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:24.738529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.738390 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" event={"ID":"af4e2767-65a2-4f53-b23a-673b34dfd0ad","Type":"ContainerStarted","Data":"233310a23bd9c3f054aa5128e0d4b2aa570f39b9f99287e132792926f57f858d"} Nov 04 01:31:24.740172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.740101 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" event={"ID":"17f25302-ecbc-4091-86d9-19234028c089","Type":"ContainerStarted","Data":"9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265"} Nov 04 01:31:24.740287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.740201 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" event={"ID":"17f25302-ecbc-4091-86d9-19234028c089","Type":"ContainerStarted","Data":"e5fab44523f5ef9e955539e73d4ac14010cb34765dd4e5c6d2504ef4218ab9d4"} Nov 04 01:31:24.740287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.740234 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:24.768875 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.768803 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" podStartSLOduration=1.768785878 podStartE2EDuration="1.768785878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:24.761604116 +0000 UTC m=+2023.733353010" watchObservedRunningTime="2026-11-04 01:31:24.768785878 +0000 UTC m=+2023.740534757" Nov 04 01:31:24.782923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:24.782824 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" podStartSLOduration=1.782809461 podStartE2EDuration="1.782809461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:24.780953501 +0000 UTC m=+2023.752702373" watchObservedRunningTime="2026-11-04 01:31:24.782809461 +0000 UTC m=+2023.754558335" Nov 04 01:31:28.733147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:28.732347 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp"] Nov 04 01:31:28.733147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:28.732617 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" containerID="cri-o://ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d" gracePeriod=30 Nov 04 01:31:28.812146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:28.803377 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl"] Nov 04 01:31:28.812146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:28.803607 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" containerID="cri-o://d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae" gracePeriod=30 Nov 04 01:31:29.919950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:29.919907 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:31:29.996030 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:29.995984 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:31:30.148925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:30.147435 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:31:30.469256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:30.469201 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:31:30.954143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:30.951605 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:31:30.954143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:30.952190 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:31:31.009616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.007239 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:31:31.009616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.008048 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:31:31.153871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.153240 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:31:31.194146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.182019 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:31:31.194146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.191781 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:31:31.404144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.403592 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Nov 04 01:31:31.611143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.602818 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:31:31.611143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.602883 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:31:31.611143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.602919 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:31:31.647167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.647049 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:31:32.001144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:31.998665 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:31:32.421199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.412167 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:31:32.477220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.470432 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:31:32.534253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.534210 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6"] Nov 04 01:31:32.535641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.535594 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" containerID="cri-o://e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25" gracePeriod=30 Nov 04 01:31:32.602227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.602190 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6"] Nov 04 01:31:32.602978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.602691 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" containerID="cri-o://cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85" gracePeriod=30 Nov 04 01:31:32.641009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.640241 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg"] Nov 04 01:31:32.641009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.640459 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" containerID="cri-o://0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a" gracePeriod=30 Nov 04 01:31:32.697814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.697715 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69"] Nov 04 01:31:32.711151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.711110 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.711492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.711465 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69"] Nov 04 01:31:32.719635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.719603 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:31:32.719829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.719808 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:31:32.719947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.719861 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:31:32.730698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.730670 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:31:32.776135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.776083 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvpp\" (UniqueName: \"kubernetes.io/projected/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-kube-api-access-zbvpp\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.776325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.776269 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.776325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.776306 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.861513 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.861479 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89"] Nov 04 01:31:32.867764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.867739 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.877584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.877558 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.877712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.877599 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-socket\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.877712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.877640 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-token\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.877712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.877701 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8pqw\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-kube-api-access-x8pqw\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.877878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.877731 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istiod-ca-cert\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.877878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.877856 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvpp\" (UniqueName: \"kubernetes.io/projected/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-kube-api-access-zbvpp\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.877984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.877889 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-data\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.877984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.877930 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-podinfo\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.878089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.878002 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-certs\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.878089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.878052 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-credential-socket\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.878089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.878072 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-envoy\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.878253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.878163 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.878967 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.878945 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.881672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.881637 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.889996 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.889751 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-920689-openshift-default-dockercfg-5nx7z\"" Nov 04 01:31:32.896106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.896080 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89"] Nov 04 01:31:32.907242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.907216 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvpp\" (UniqueName: \"kubernetes.io/projected/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-kube-api-access-zbvpp\") pod \"maas-api-revoke-sub-32cc1541-m2c69\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:32.979400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979327 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-data\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.979400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979392 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-podinfo\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.979610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979465 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-certs\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.979610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979537 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-credential-socket\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.979610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979564 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-envoy\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.979803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979628 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-socket\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.979803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979689 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-token\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.979803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979784 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8pqw\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-kube-api-access-x8pqw\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.979803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979793 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-data\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.980646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.979831 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istiod-ca-cert\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.980646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.980091 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-socket\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.980646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.980114 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-credential-socket\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.980646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.980269 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-certs\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.980877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.980693 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istiod-ca-cert\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.982449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.982423 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-envoy\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.983028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.982570 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-podinfo\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.995227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.995180 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8pqw\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-kube-api-access-x8pqw\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:32.996789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:32.996748 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-token\") pod \"e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:33.034559 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:33.033577 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:33.200435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:33.200396 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:33.710389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:33.710293 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69"] Nov 04 01:31:33.716045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:33.716015 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89"] Nov 04 01:31:33.717796 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:33.717758 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848d3e60_4a1e_438c_b5ad_b348c2f9f4b9.slice/crio-38857babc35bbef3c8ff74bb3c7982ca3f9aeea514f23689b1a202165697b3c5 WatchSource:0}: Error finding container 38857babc35bbef3c8ff74bb3c7982ca3f9aeea514f23689b1a202165697b3c5: Status 404 returned error can't find the container with id 38857babc35bbef3c8ff74bb3c7982ca3f9aeea514f23689b1a202165697b3c5 Nov 04 01:31:33.718612 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:33.718588 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f13c29_48fe_41ce_ba7f_5bfe6d073eab.slice/crio-f537ba5cd65820db24f4c03c371297cabdf47b6f5ed38bcd8b9a3096a828a4a8 WatchSource:0}: Error finding container f537ba5cd65820db24f4c03c371297cabdf47b6f5ed38bcd8b9a3096a828a4a8: Status 404 returned error can't find the container with id f537ba5cd65820db24f4c03c371297cabdf47b6f5ed38bcd8b9a3096a828a4a8 Nov 04 01:31:33.721532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:33.721500 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:33.721641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:33.721564 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:33.721641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:33.721594 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:31:34.113740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.113348 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:31:34.122957 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:34.122889 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:31:34.257789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.257751 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-certs\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.257926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.257815 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istiod-ca-cert\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.257926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.257876 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-data\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.257926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.257921 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-podinfo\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.258082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.257995 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-envoy\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.258474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.258286 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-token\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-token\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.258474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.258344 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-socket\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.258474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.258429 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-credential-socket\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.258474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.258461 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istiod-ca-cert" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:34.258677 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.258587 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-kube-api-access-2ndwq\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-kube-api-access-2ndwq\") pod \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\" (UID: \"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa\") " Nov 04 01:31:34.258822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.258786 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-certs" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.259074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.259054 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-data" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.259588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.259556 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.259588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.259586 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.259744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.259600 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.260174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.260093 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-credential-socket" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.260369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.260338 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-socket" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.262354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.261825 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:31:34.262354 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.262322 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-kube-api-access-2ndwq" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "kube-api-access-2ndwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:34.262517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.262494 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-envoy" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.262774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.262730 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-podinfo" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:34.266376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.266329 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-token" pod "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" (UID: "bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:34.360813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.360728 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-token\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-token\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.360964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.360909 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528f8e10-c023-4ed5-a75a-fed43dbadae7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f8e10-c023-4ed5-a75a-fed43dbadae7-istiod-ca-cert\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.361092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.361076 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-socket\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.361173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.361157 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-credential-socket\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.361221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.361207 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-certs\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.361260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.361241 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-podinfo\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.361325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.361311 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-data\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.361393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.361379 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-envoy\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.361480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.361453 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-kube-api-access-qfg7k\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-kube-api-access-qfg7k\") pod \"528f8e10-c023-4ed5-a75a-fed43dbadae7\" (UID: \"528f8e10-c023-4ed5-a75a-fed43dbadae7\") " Nov 04 01:31:34.362377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.362356 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndwq\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-kube-api-access-2ndwq\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.362425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.362385 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.362425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.362400 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.362425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.362414 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.362526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.362429 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.362526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.362443 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.363298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.363270 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-socket" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.363514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.363296 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-data" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.363906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.363882 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-credential-socket" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.364099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.364082 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528f8e10-c023-4ed5-a75a-fed43dbadae7-istiod-ca-cert" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:34.364412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.364390 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-certs" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.365463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.365440 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-podinfo" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:34.365683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.365646 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-token" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:34.365766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.365682 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-envoy" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:34.365937 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.365917 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-kube-api-access-qfg7k" pod "528f8e10-c023-4ed5-a75a-fed43dbadae7" (UID: "528f8e10-c023-4ed5-a75a-fed43dbadae7"). InnerVolumeSpecName "kube-api-access-qfg7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:34.525243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.482161 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfg7k\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-kube-api-access-qfg7k\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.525500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.525476 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.525612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.525505 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/528f8e10-c023-4ed5-a75a-fed43dbadae7-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.525612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.525521 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.525612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.525536 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.525612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.525556 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.525612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.525571 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.525612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.525585 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.525612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.525601 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/528f8e10-c023-4ed5-a75a-fed43dbadae7-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:34.575440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.575398 5051 generic.go:356] "Generic (PLEG): container finished" podID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerID="ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d" exitCode=0 Nov 04 01:31:34.575883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.575469 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" event={"ID":"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa","Type":"ContainerDied","Data":"ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d"} Nov 04 01:31:34.576001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.575921 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" event={"ID":"bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa","Type":"ContainerDied","Data":"6c547a309da32ba193e68bc2b6e5f44d6bc1ffe1267d78e79ea326abbaeddedb"} Nov 04 01:31:34.576001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.575947 5051 scope.go:122] "RemoveContainer" containerID="ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d" Nov 04 01:31:34.576100 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.575537 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp" Nov 04 01:31:34.578998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.578969 5051 generic.go:356] "Generic (PLEG): container finished" podID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerID="d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae" exitCode=0 Nov 04 01:31:34.579371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.579349 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" event={"ID":"528f8e10-c023-4ed5-a75a-fed43dbadae7","Type":"ContainerDied","Data":"d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae"} Nov 04 01:31:34.579436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.579404 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" event={"ID":"528f8e10-c023-4ed5-a75a-fed43dbadae7","Type":"ContainerDied","Data":"8585806518af2ac7a4103a9ca7d945b0713386cab357e6688e944452541ef716"} Nov 04 01:31:34.579511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.579498 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl" Nov 04 01:31:34.586922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.586901 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" event={"ID":"01f13c29-48fe-41ce-ba7f-5bfe6d073eab","Type":"ContainerStarted","Data":"3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93"} Nov 04 01:31:34.587686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.587664 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" event={"ID":"01f13c29-48fe-41ce-ba7f-5bfe6d073eab","Type":"ContainerStarted","Data":"f537ba5cd65820db24f4c03c371297cabdf47b6f5ed38bcd8b9a3096a828a4a8"} Nov 04 01:31:34.590999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.590902 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" event={"ID":"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9","Type":"ContainerStarted","Data":"f58e5ae022688ee9638728632ed6520f2a7ac86cf6f691f51538c9402d55d4b1"} Nov 04 01:31:34.591926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.591910 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" event={"ID":"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9","Type":"ContainerStarted","Data":"38857babc35bbef3c8ff74bb3c7982ca3f9aeea514f23689b1a202165697b3c5"} Nov 04 01:31:34.594302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.594248 5051 scope.go:122] "RemoveContainer" containerID="ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d" Nov 04 01:31:34.594516 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:34.594493 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d\": container with ID starting with ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d not found: ID does not exist" containerID="ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d" Nov 04 01:31:34.594655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.594525 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d"} err="failed to get container status \"ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d\": rpc error: code = NotFound desc = could not find container \"ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d\": container with ID starting with ebc266289a1c9bf137aa1aafd15f87e46ecb1426ad4a98b76d1a389e2d40570d not found: ID does not exist" Nov 04 01:31:34.594655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.594547 5051 scope.go:122] "RemoveContainer" containerID="d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae" Nov 04 01:31:34.615179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.615157 5051 scope.go:122] "RemoveContainer" containerID="d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae" Nov 04 01:31:34.615463 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:34.615439 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae\": container with ID starting with d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae not found: ID does not exist" containerID="d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae" Nov 04 01:31:34.615565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.615468 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae"} err="failed to get container status \"d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae\": rpc error: code = NotFound desc = could not find container \"d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae\": container with ID starting with d5a8c44c8b28d2459159682e01e1a1940f475e802d7b8d1f080c9435490d54ae not found: ID does not exist" Nov 04 01:31:34.616141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.616075 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podStartSLOduration=2.616057518 podStartE2EDuration="2.616057518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:34.608612384 +0000 UTC m=+2033.580361255" watchObservedRunningTime="2026-11-04 01:31:34.616057518 +0000 UTC m=+2033.587806391" Nov 04 01:31:34.628953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.628900 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" podStartSLOduration=2.628882011 podStartE2EDuration="2.628882011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:34.625604515 +0000 UTC m=+2033.597353385" watchObservedRunningTime="2026-11-04 01:31:34.628882011 +0000 UTC m=+2033.600630882" Nov 04 01:31:34.643022 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.642993 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl"] Nov 04 01:31:34.647580 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.647555 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-k9ghl"] Nov 04 01:31:34.667142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.667092 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp"] Nov 04 01:31:34.671713 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:34.671686 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc23826f_d7b7_403a_ad60_e4a5f1f0b0aa.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc23826f_d7b7_403a_ad60_e4a5f1f0b0aa.slice/crio-6c547a309da32ba193e68bc2b6e5f44d6bc1ffe1267d78e79ea326abbaeddedb\": RecentStats: unable to find data in memory cache]" Nov 04 01:31:34.674605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:34.674578 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-mhcsp"] Nov 04 01:31:35.316709 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.316671 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" path="/var/lib/kubelet/pods/528f8e10-c023-4ed5-a75a-fed43dbadae7/volumes" Nov 04 01:31:35.318388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.318103 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" path="/var/lib/kubelet/pods/bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa/volumes" Nov 04 01:31:35.319327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.319308 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:35.319603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.319543 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:35.607803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.607706 5051 generic.go:356] "Generic (PLEG): container finished" podID="848d3e60-4a1e-438c-b5ad-b348c2f9f4b9" containerID="f58e5ae022688ee9638728632ed6520f2a7ac86cf6f691f51538c9402d55d4b1" exitCode=0 Nov 04 01:31:35.610359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.610324 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" event={"ID":"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9","Type":"ContainerDied","Data":"f58e5ae022688ee9638728632ed6520f2a7ac86cf6f691f51538c9402d55d4b1"} Nov 04 01:31:35.615859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.615829 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:35.617199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.617138 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:31:35.826069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.826039 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:31:35.827924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:35.827901 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:31:37.182671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.182644 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:37.261134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.261028 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-openshift-ca\") pod \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " Nov 04 01:31:37.261309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.261261 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-kube-api-access-zbvpp\" (UniqueName: \"kubernetes.io/projected/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-kube-api-access-zbvpp\") pod \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " Nov 04 01:31:37.261371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.261335 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-xks-ca\") pod \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\" (UID: \"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9\") " Nov 04 01:31:37.261873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.261847 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-openshift-ca" pod "848d3e60-4a1e-438c-b5ad-b348c2f9f4b9" (UID: "848d3e60-4a1e-438c-b5ad-b348c2f9f4b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:37.262573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.262549 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.266594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.266558 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-kube-api-access-zbvpp" pod "848d3e60-4a1e-438c-b5ad-b348c2f9f4b9" (UID: "848d3e60-4a1e-438c-b5ad-b348c2f9f4b9"). InnerVolumeSpecName "kube-api-access-zbvpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:37.267102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.267072 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-xks-ca" pod "848d3e60-4a1e-438c-b5ad-b348c2f9f4b9" (UID: "848d3e60-4a1e-438c-b5ad-b348c2f9f4b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:31:37.365615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.365518 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbvpp\" (UniqueName: \"kubernetes.io/projected/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-kube-api-access-zbvpp\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.365615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.365557 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:37.634019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.633945 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" Nov 04 01:31:37.634019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.633943 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69" event={"ID":"848d3e60-4a1e-438c-b5ad-b348c2f9f4b9","Type":"ContainerDied","Data":"38857babc35bbef3c8ff74bb3c7982ca3f9aeea514f23689b1a202165697b3c5"} Nov 04 01:31:37.634307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:37.634028 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38857babc35bbef3c8ff74bb3c7982ca3f9aeea514f23689b1a202165697b3c5" Nov 04 01:31:38.046872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.046842 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:31:38.133286 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133253 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-kube-api-access-9kgp8\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-kube-api-access-9kgp8\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.133475 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133459 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-credential-socket\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.133564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133503 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-envoy\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.133627 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133576 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istiod-ca-cert\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.133692 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133678 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-podinfo\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.133747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133716 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-token\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-token\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.133853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133835 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-data\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.133939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133907 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-credential-socket" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.134002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.133937 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-certs\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.134106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.134089 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-socket\") pod \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\" (UID: \"05cdea56-609e-4f8e-b99d-37d1e5b7f916\") " Nov 04 01:31:38.135065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.135038 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.135364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.135338 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-socket" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.136799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.136770 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-kube-api-access-9kgp8" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "kube-api-access-9kgp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:38.137050 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.137016 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-podinfo" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:38.137469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.137442 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istiod-ca-cert" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:38.137729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.137704 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-token" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:38.137813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.137734 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-data" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.137979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.137958 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-certs" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.138990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.138838 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-envoy" pod "05cdea56-609e-4f8e-b99d-37d1e5b7f916" (UID: "05cdea56-609e-4f8e-b99d-37d1e5b7f916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.241625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.241341 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.241625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.241372 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.241625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.241387 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.241625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.241400 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.241625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.241416 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.241625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.241430 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.241625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.241446 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05cdea56-609e-4f8e-b99d-37d1e5b7f916-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.241625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.241461 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgp8\" (UniqueName: \"kubernetes.io/projected/05cdea56-609e-4f8e-b99d-37d1e5b7f916-kube-api-access-9kgp8\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.411155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.411115 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:31:38.446460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.446434 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istiod-ca-cert\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.446562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.446503 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-envoy\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.446562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.446545 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-token\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-token\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.446687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.446582 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-credential-socket\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.447306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.447265 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-data\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.447404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.447347 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-certs\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.447455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.447423 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-socket\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.447500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.447482 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-podinfo\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.447639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.447621 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-kube-api-access-cb8qs\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-kube-api-access-cb8qs\") pod \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\" (UID: \"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea\") " Nov 04 01:31:38.449554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.449367 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-credential-socket" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.450076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.449814 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istiod-ca-cert" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:38.451433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.450799 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-socket" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.451433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.451102 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-data" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.452263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.451942 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr"] Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453372 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453397 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453425 5051 cpu_manager.go:414] "containerMap: removing container" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453433 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453448 5051 cpu_manager.go:414] "containerMap: removing container" podUID="848d3e60-4a1e-438c-b5ad-b348c2f9f4b9" containerName="revoke-keys" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453468 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848d3e60-4a1e-438c-b5ad-b348c2f9f4b9" containerName="revoke-keys" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453533 5051 cpu_manager.go:414] "containerMap: removing container" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453541 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453572 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" Nov 04 01:31:38.454387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.453580 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" Nov 04 01:31:38.457606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.455108 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerName="istio-proxy" Nov 04 01:31:38.457606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.455160 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc23826f-d7b7-403a-ad60-e4a5f1f0b0aa" containerName="istio-proxy" Nov 04 01:31:38.457606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.455177 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerName="istio-proxy" Nov 04 01:31:38.457606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.455190 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="848d3e60-4a1e-438c-b5ad-b348c2f9f4b9" containerName="revoke-keys" Nov 04 01:31:38.457606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.455215 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="528f8e10-c023-4ed5-a75a-fed43dbadae7" containerName="istio-proxy" Nov 04 01:31:38.460973 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.458382 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-kube-api-access-cb8qs" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "kube-api-access-cb8qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:38.469418 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.469377 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-certs" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.471504 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.471474 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:38.472044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.472005 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-podinfo" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:38.472364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.472340 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-envoy" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.472861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.472827 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-token" pod "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" (UID: "c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:38.479283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.479168 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-920689\"" Nov 04 01:31:38.484885 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.483484 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-920689-dockercfg-k2c9x\"" Nov 04 01:31:38.486081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.486061 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr"] Nov 04 01:31:38.494734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.494715 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:31:38.518645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.510781 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44"] Nov 04 01:31:38.518645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.513148 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" Nov 04 01:31:38.518645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.513169 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" Nov 04 01:31:38.518645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.513288 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" Nov 04 01:31:38.518645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.513298 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" Nov 04 01:31:38.518645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.513888 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" Nov 04 01:31:38.518645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.513926 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" containerName="istio-proxy" Nov 04 01:31:38.557750 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.557710 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-socket\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.709744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.574226 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-socket" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.742258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.697780 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1c36e46-07ec-47df-a65c-d35233518e01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c36e46-07ec-47df-a65c-d35233518e01-istio-podinfo\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.742540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.742515 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-kube-api-access-6qvph\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-kube-api-access-6qvph\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.742738 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.742721 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-istio-token\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-istio-token\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.742873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.742858 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1c36e46-07ec-47df-a65c-d35233518e01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c36e46-07ec-47df-a65c-d35233518e01-istiod-ca-cert\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.743019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.743004 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-credential-socket\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.747141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.743325 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-envoy\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.747343 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.747321 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-certs\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.775969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.768377 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-data\") pod \"e1c36e46-07ec-47df-a65c-d35233518e01\" (UID: \"e1c36e46-07ec-47df-a65c-d35233518e01\") " Nov 04 01:31:38.775969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.771698 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44"] Nov 04 01:31:38.775969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.771867 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:38.855264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855216 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46690fdf-fd41-4366-b213-a60d37e78b45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr\" (UID: \"46690fdf-fd41-4366-b213-a60d37e78b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:38.855481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855458 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxfrj\" (UniqueName: \"kubernetes.io/projected/46690fdf-fd41-4366-b213-a60d37e78b45-kube-api-access-nxfrj\") pod \"payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr\" (UID: \"46690fdf-fd41-4366-b213-a60d37e78b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:38.855811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855787 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8qs\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-kube-api-access-cb8qs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.855943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855814 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.855943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855838 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.855943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855855 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.855943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855869 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.855943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855886 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.855943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855908 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.855943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855922 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.855943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855939 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.856400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.855954 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.887143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.768302 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-certs" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.889046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.817023 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c36e46-07ec-47df-a65c-d35233518e01-istiod-ca-cert" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:31:38.889260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.848624 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1c36e46-07ec-47df-a65c-d35233518e01-istio-podinfo" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:31:38.898497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.867133 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-credential-socket" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.899111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.877459 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-data" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.942767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.942720 5051 generic.go:356] "Generic (PLEG): container finished" podID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" containerID="0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a" exitCode=0 Nov 04 01:31:38.943000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.942955 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" event={"ID":"05cdea56-609e-4f8e-b99d-37d1e5b7f916","Type":"ContainerDied","Data":"0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a"} Nov 04 01:31:38.943081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.943026 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" event={"ID":"05cdea56-609e-4f8e-b99d-37d1e5b7f916","Type":"ContainerDied","Data":"59bcd37f404a89e8fd3fe9ebf7edb8dfcbce4b412cb901b5f4cd73563ec28589"} Nov 04 01:31:38.943177 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.943065 5051 scope.go:122] "RemoveContainer" containerID="0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a" Nov 04 01:31:38.944718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.944229 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-kube-api-access-6qvph" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "kube-api-access-6qvph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:38.944718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.944531 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-istio-token" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:31:38.947195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.947171 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg" Nov 04 01:31:38.947436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.947410 5051 generic.go:356] "Generic (PLEG): container finished" podID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" containerID="e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25" exitCode=0 Nov 04 01:31:38.948030 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.948006 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" event={"ID":"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea","Type":"ContainerDied","Data":"e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25"} Nov 04 01:31:38.948186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.948169 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" event={"ID":"c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea","Type":"ContainerDied","Data":"adede04078a8b42567a22398f3bf748c5010410b0992b1861e27c8b8c51385d9"} Nov 04 01:31:38.948638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.948619 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6" Nov 04 01:31:38.951971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.951933 5051 generic.go:356] "Generic (PLEG): container finished" podID="e1c36e46-07ec-47df-a65c-d35233518e01" containerID="cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85" exitCode=0 Nov 04 01:31:38.952252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.952220 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" event={"ID":"e1c36e46-07ec-47df-a65c-d35233518e01","Type":"ContainerDied","Data":"cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85"} Nov 04 01:31:38.952448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.952419 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" event={"ID":"e1c36e46-07ec-47df-a65c-d35233518e01","Type":"ContainerDied","Data":"2f84f2620289143ed702edc53a3d484f6c7ebd240b3d7be9c9a7bb904eeda04c"} Nov 04 01:31:38.952699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.952671 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6" Nov 04 01:31:38.955414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.955391 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-envoy" pod "e1c36e46-07ec-47df-a65c-d35233518e01" (UID: "e1c36e46-07ec-47df-a65c-d35233518e01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964238 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6zx\" (UniqueName: \"kubernetes.io/projected/77f531bf-f087-4ccc-a582-91a0a87e04a7-kube-api-access-ll6zx\") pod \"payload-processing-e2e-api-a-920689-6789748bf4-nks44\" (UID: \"77f531bf-f087-4ccc-a582-91a0a87e04a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964365 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46690fdf-fd41-4366-b213-a60d37e78b45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr\" (UID: \"46690fdf-fd41-4366-b213-a60d37e78b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964475 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxfrj\" (UniqueName: \"kubernetes.io/projected/46690fdf-fd41-4366-b213-a60d37e78b45-kube-api-access-nxfrj\") pod \"payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr\" (UID: \"46690fdf-fd41-4366-b213-a60d37e78b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964594 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f531bf-f087-4ccc-a582-91a0a87e04a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-920689-6789748bf4-nks44\" (UID: \"77f531bf-f087-4ccc-a582-91a0a87e04a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964821 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964838 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964852 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964865 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c36e46-07ec-47df-a65c-d35233518e01-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964883 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c36e46-07ec-47df-a65c-d35233518e01-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964897 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvph\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-kube-api-access-6qvph\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964912 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c36e46-07ec-47df-a65c-d35233518e01-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.964929 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c36e46-07ec-47df-a65c-d35233518e01-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:31:38.967148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.965825 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46690fdf-fd41-4366-b213-a60d37e78b45-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr\" (UID: \"46690fdf-fd41-4366-b213-a60d37e78b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:38.978683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.978577 5051 scope.go:122] "RemoveContainer" containerID="0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a" Nov 04 01:31:38.979097 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:38.979068 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a\": container with ID starting with 0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a not found: ID does not exist" containerID="0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a" Nov 04 01:31:38.982505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.979281 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a"} err="failed to get container status \"0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a\": rpc error: code = NotFound desc = could not find container \"0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a\": container with ID starting with 0d95ca2436beaf4833ec2ce35ddfa94eade6778c0803241b49732846c7ae7d6a not found: ID does not exist" Nov 04 01:31:38.982505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.982239 5051 scope.go:122] "RemoveContainer" containerID="e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25" Nov 04 01:31:38.984551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.984525 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxfrj\" (UniqueName: \"kubernetes.io/projected/46690fdf-fd41-4366-b213-a60d37e78b45-kube-api-access-nxfrj\") pod \"payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr\" (UID: \"46690fdf-fd41-4366-b213-a60d37e78b45\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:38.999206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.999184 5051 scope.go:122] "RemoveContainer" containerID="e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25" Nov 04 01:31:38.999686 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:38.999539 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25\": container with ID starting with e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25 not found: ID does not exist" containerID="e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25" Nov 04 01:31:38.999686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.999569 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25"} err="failed to get container status \"e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25\": rpc error: code = NotFound desc = could not find container \"e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25\": container with ID starting with e8492ec3e7ab45b8f8d1b3f3f3ee679850937a75d40bcfe099058e2ec8ff5e25 not found: ID does not exist" Nov 04 01:31:38.999686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:38.999592 5051 scope.go:122] "RemoveContainer" containerID="cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85" Nov 04 01:31:39.003716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.003695 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg"] Nov 04 01:31:39.010963 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.010942 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-ht6qg"] Nov 04 01:31:39.013778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.013757 5051 scope.go:122] "RemoveContainer" containerID="45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a" Nov 04 01:31:39.071148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.068221 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f531bf-f087-4ccc-a582-91a0a87e04a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-920689-6789748bf4-nks44\" (UID: \"77f531bf-f087-4ccc-a582-91a0a87e04a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:39.071148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.068423 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6zx\" (UniqueName: \"kubernetes.io/projected/77f531bf-f087-4ccc-a582-91a0a87e04a7-kube-api-access-ll6zx\") pod \"payload-processing-e2e-api-a-920689-6789748bf4-nks44\" (UID: \"77f531bf-f087-4ccc-a582-91a0a87e04a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:39.115282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.100233 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f531bf-f087-4ccc-a582-91a0a87e04a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-920689-6789748bf4-nks44\" (UID: \"77f531bf-f087-4ccc-a582-91a0a87e04a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:39.129569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.129533 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:39.132258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.132231 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6zx\" (UniqueName: \"kubernetes.io/projected/77f531bf-f087-4ccc-a582-91a0a87e04a7-kube-api-access-ll6zx\") pod \"payload-processing-e2e-api-a-920689-6789748bf4-nks44\" (UID: \"77f531bf-f087-4ccc-a582-91a0a87e04a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:39.144224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.144196 5051 scope.go:122] "RemoveContainer" containerID="cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85" Nov 04 01:31:39.144865 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:39.144841 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85\": container with ID starting with cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85 not found: ID does not exist" containerID="cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85" Nov 04 01:31:39.145105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.145078 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85"} err="failed to get container status \"cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85\": rpc error: code = NotFound desc = could not find container \"cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85\": container with ID starting with cfbc6cbbcfaaec16674d4143764990c86c254ec1215bc6bbe924f0edade55d85 not found: ID does not exist" Nov 04 01:31:39.145256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.145242 5051 scope.go:122] "RemoveContainer" containerID="45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a" Nov 04 01:31:39.145399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.145244 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6"] Nov 04 01:31:39.148338 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:31:39.147997 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a\": container with ID starting with 45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a not found: ID does not exist" containerID="45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a" Nov 04 01:31:39.148338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.148028 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a"} err="failed to get container status \"45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a\": rpc error: code = NotFound desc = could not find container \"45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a\": container with ID starting with 45d4f56e815f96a5c53e1dec523de1e105fd4052b4ad39ba2f7d3c2c81726e0a not found: ID does not exist" Nov 04 01:31:39.148338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.148266 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-z6fk6"] Nov 04 01:31:39.195548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.195519 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cdea56-609e-4f8e-b99d-37d1e5b7f916" path="/var/lib/kubelet/pods/05cdea56-609e-4f8e-b99d-37d1e5b7f916/volumes" Nov 04 01:31:39.196515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.196493 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea" path="/var/lib/kubelet/pods/c4ac22d4-8a67-457f-b0b7-fee2d9ee93ea/volumes" Nov 04 01:31:39.273437 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.273407 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:39.276386 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.276356 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6"] Nov 04 01:31:39.283224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.283167 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df6da4c0-openshift-default-764b65fbbf-kpjq6"] Nov 04 01:31:39.461187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.461157 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr"] Nov 04 01:31:39.466350 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:39.465776 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46690fdf_fd41_4366_b213_a60d37e78b45.slice/crio-6ad6724d9f9c6197cbbbe8361b9f8ed6b9f19f801e2d5d2ab17706c0ef3c23f0 WatchSource:0}: Error finding container 6ad6724d9f9c6197cbbbe8361b9f8ed6b9f19f801e2d5d2ab17706c0ef3c23f0: Status 404 returned error can't find the container with id 6ad6724d9f9c6197cbbbe8361b9f8ed6b9f19f801e2d5d2ab17706c0ef3c23f0 Nov 04 01:31:39.582588 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:31:39.582557 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f531bf_f087_4ccc_a582_91a0a87e04a7.slice/crio-407f0b21ae34a5de083aa221cc4f21614be67642b628265f4ddec63e31170cde WatchSource:0}: Error finding container 407f0b21ae34a5de083aa221cc4f21614be67642b628265f4ddec63e31170cde: Status 404 returned error can't find the container with id 407f0b21ae34a5de083aa221cc4f21614be67642b628265f4ddec63e31170cde Nov 04 01:31:39.582747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.582722 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44"] Nov 04 01:31:39.969503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.969415 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" event={"ID":"77f531bf-f087-4ccc-a582-91a0a87e04a7","Type":"ContainerStarted","Data":"ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea"} Nov 04 01:31:39.969503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.969469 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" event={"ID":"77f531bf-f087-4ccc-a582-91a0a87e04a7","Type":"ContainerStarted","Data":"407f0b21ae34a5de083aa221cc4f21614be67642b628265f4ddec63e31170cde"} Nov 04 01:31:39.969752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.969570 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:39.972219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.972193 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" event={"ID":"46690fdf-fd41-4366-b213-a60d37e78b45","Type":"ContainerStarted","Data":"4fda571039a8f949c7bf414f33a8cbfadd28ca0f62620604d3750e018fc78a12"} Nov 04 01:31:39.972356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.972235 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" event={"ID":"46690fdf-fd41-4366-b213-a60d37e78b45","Type":"ContainerStarted","Data":"6ad6724d9f9c6197cbbbe8361b9f8ed6b9f19f801e2d5d2ab17706c0ef3c23f0"} Nov 04 01:31:39.972356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.972304 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:39.992077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:39.992026 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" podStartSLOduration=1.9920033080000001 podStartE2EDuration="1.992003308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:39.986909227 +0000 UTC m=+2038.958658103" watchObservedRunningTime="2026-11-04 01:31:39.992003308 +0000 UTC m=+2038.963752182" Nov 04 01:31:40.013385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:40.013343 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" podStartSLOduration=2.013330746 podStartE2EDuration="2.013330746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:31:40.008642306 +0000 UTC m=+2038.980391189" watchObservedRunningTime="2026-11-04 01:31:40.013330746 +0000 UTC m=+2038.985079616" Nov 04 01:31:40.040942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:40.040913 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp"] Nov 04 01:31:40.045191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:40.045169 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e067977e-2lpqp"] Nov 04 01:31:41.204834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:41.204798 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72b5492-f761-46d0-aa80-2c9abb6b0b81" path="/var/lib/kubelet/pods/c72b5492-f761-46d0-aa80-2c9abb6b0b81/volumes" Nov 04 01:31:41.205652 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:41.205617 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c36e46-07ec-47df-a65c-d35233518e01" path="/var/lib/kubelet/pods/e1c36e46-07ec-47df-a65c-d35233518e01/volumes" Nov 04 01:31:47.865842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:47.865803 5051 scope.go:122] "RemoveContainer" containerID="a0d3595ba13e6d64fa596be1e9b7a18a79b1043f88d84a6f87acbdecae4c915a" Nov 04 01:31:47.879555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:47.879529 5051 scope.go:122] "RemoveContainer" containerID="25f7e6064d4882603f005ab167c36cf7b67488de00831be1d98ab26d7173841c" Nov 04 01:31:47.897264 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:47.897236 5051 scope.go:122] "RemoveContainer" containerID="747858deade6e3a7f8a24e63e098427d810643138aaf11cd755f8d07ec13cc51" Nov 04 01:31:47.908407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:47.908386 5051 scope.go:122] "RemoveContainer" containerID="ecfad41ddb655cd9bc43c6c51e9e7d2f5fc0b76f3259094afd5a304f4f87185e" Nov 04 01:31:47.918724 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:47.918704 5051 scope.go:122] "RemoveContainer" containerID="5e09dd83f2cc613c0121b10a01c8dd41407cdded64d054e1a367123fcacb70af" Nov 04 01:31:51.065971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:51.065937 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:31:51.101151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:51.091203 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:31:59.278142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.271807 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk"] Nov 04 01:31:59.433115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.432183 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk"] Nov 04 01:31:59.450308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.434097 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.450308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.445359 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c75ea1-openshift-default-dockercfg-7g542\"" Nov 04 01:31:59.606191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.605861 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.606191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.605937 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.606191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.606010 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.606191 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.606170 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdgnt\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-kube-api-access-pdgnt\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.606537 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.606252 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.606537 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.606447 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.606644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.606583 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.606644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.606622 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.606745 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.606721 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12db2eea-fca9-45ab-b564-a7289c92fe5d-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708103 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708215 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdgnt\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-kube-api-access-pdgnt\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708263 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708349 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708435 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708460 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708517 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12db2eea-fca9-45ab-b564-a7289c92fe5d-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708576 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.708797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708604 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.709368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.708997 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-data\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.709702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.709667 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.709988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.709963 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-credential-socket\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.710280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.710255 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-certs\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.710696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.710671 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12db2eea-fca9-45ab-b564-a7289c92fe5d-istiod-ca-cert\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.712334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.712312 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-envoy\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.717565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.717543 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-podinfo\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.736150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.728252 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-token\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.750578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.750329 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdgnt\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-kube-api-access-pdgnt\") pod \"e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:31:59.887189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:31:59.887093 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:32:00.186723 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:32:00.186691 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12db2eea_fca9_45ab_b564_a7289c92fe5d.slice/crio-554a3e5829a80f04d4f4dbb5cd21ffaa2bf98ce89082f67afde7f297c2eaaa26 WatchSource:0}: Error finding container 554a3e5829a80f04d4f4dbb5cd21ffaa2bf98ce89082f67afde7f297c2eaaa26: Status 404 returned error can't find the container with id 554a3e5829a80f04d4f4dbb5cd21ffaa2bf98ce89082f67afde7f297c2eaaa26 Nov 04 01:32:00.188392 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:00.188368 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk"] Nov 04 01:32:00.190363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:00.190324 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:00.190474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:00.190406 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:00.190474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:00.190452 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:00.662455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:00.662416 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" event={"ID":"12db2eea-fca9-45ab-b564-a7289c92fe5d","Type":"ContainerStarted","Data":"fb797ed90ff8cea237bf18ed4abcd468d115d8689244c0e3ef1f802ff1c896a5"} Nov 04 01:32:00.664337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:00.664308 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" event={"ID":"12db2eea-fca9-45ab-b564-a7289c92fe5d","Type":"ContainerStarted","Data":"554a3e5829a80f04d4f4dbb5cd21ffaa2bf98ce89082f67afde7f297c2eaaa26"} Nov 04 01:32:00.705140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:00.704557 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" podStartSLOduration=1.704535378 podStartE2EDuration="1.704535378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:00.694519419 +0000 UTC m=+2059.666268301" watchObservedRunningTime="2026-11-04 01:32:00.704535378 +0000 UTC m=+2059.676284251" Nov 04 01:32:00.890200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:00.889535 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:32:01.821108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.819430 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt"] Nov 04 01:32:01.826234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.826210 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.839463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.839436 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-30b1a1-openshift-default-dockercfg-rbkbq\"" Nov 04 01:32:01.841772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.841745 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt"] Nov 04 01:32:01.898146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.897663 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:32:01.910925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.910889 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-data\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.911238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.911210 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-podinfo\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.911543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.911520 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-certs\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.911707 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.911691 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-envoy\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.911844 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.911826 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.911952 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.911937 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-token\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.915072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.912264 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73872956-f8af-4631-b5e3-a88f3d39f3d9-istiod-ca-cert\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.915072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.912322 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbgng\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-kube-api-access-kbgng\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:01.915072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:01.912568 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-credential-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.014650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.014617 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-certs\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.014650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.014660 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-envoy\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.014928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.014682 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.014928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.014705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-token\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.014928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.014846 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73872956-f8af-4631-b5e3-a88f3d39f3d9-istiod-ca-cert\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.015159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.015098 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-certs\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.015254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.015193 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.015549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.015531 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73872956-f8af-4631-b5e3-a88f3d39f3d9-istiod-ca-cert\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.015619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.015586 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbgng\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-kube-api-access-kbgng\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.015682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.015629 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-credential-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.015760 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.015737 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-data\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.015819 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.015807 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-podinfo\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.016039 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.016015 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-credential-socket\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.016137 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.016074 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-data\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.017171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.017146 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-envoy\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.018556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.018531 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-podinfo\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.036202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.036178 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-token\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.042026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.042000 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbgng\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-kube-api-access-kbgng\") pod \"e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.154487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.154396 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:02.486479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.485748 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt"] Nov 04 01:32:02.499878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.493516 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:02.499878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.493582 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:02.499878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.493609 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:02.720555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.719997 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" event={"ID":"73872956-f8af-4631-b5e3-a88f3d39f3d9","Type":"ContainerStarted","Data":"e3681e8a4a363778fcc523446ab34569cd3a3562430e97cbadbcc8947f47dc88"} Nov 04 01:32:02.720555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.720068 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" event={"ID":"73872956-f8af-4631-b5e3-a88f3d39f3d9","Type":"ContainerStarted","Data":"7c30b1ffd201aed341ab50902791f26a7575567a4a6d9a985bfda6d0fb7783c6"} Nov 04 01:32:02.720800 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.720772 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:32:02.721562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.721536 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:32:02.875273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:02.864479 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podStartSLOduration=1.864453844 podStartE2EDuration="1.864453844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:02.784784365 +0000 UTC m=+2061.756533238" watchObservedRunningTime="2026-11-04 01:32:02.864453844 +0000 UTC m=+2061.836202761" Nov 04 01:32:03.191644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:03.179525 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:03.192553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:03.192521 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:03.192788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:03.192745 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:04.242491 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:04.242165 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:04.255631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:04.255587 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:04.458331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:04.458267 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:04.545794 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:04.545687 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:32:05.245304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.244317 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:05.245304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.244384 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:05.467182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.446416 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:32:05.467182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.424702 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:05.467182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.461200 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:05.467182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.461693 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:05.479672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.475247 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:32:05.788173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.761204 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt"] Nov 04 01:32:05.820557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.820530 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.889644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.889592 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:05.928006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.926091 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bfc54d-openshift-default-dockercfg-vjzfx\"" Nov 04 01:32:05.958186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.958151 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5123781-6eea-46f8-8893-175421c25141-istiod-ca-cert\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.969332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.969295 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5123781-6eea-46f8-8893-175421c25141-istio-podinfo\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.969676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.969658 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-certs\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.969857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.969842 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-socket\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.969972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.969957 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-envoy\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.970082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.970070 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-istio-token\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.970352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.970331 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-data\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.970572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.970552 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-credential-socket\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:05.970707 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:05.970691 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpbm\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-kube-api-access-7xpbm\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.123383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.123286 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:06.203174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.203112 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5123781-6eea-46f8-8893-175421c25141-istio-podinfo\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.206149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.203525 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-certs\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.206445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.206420 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-socket\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.206576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.206562 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-envoy\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.206695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.206678 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-istio-token\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.220816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.220763 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-data\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.221204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.221175 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-credential-socket\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.221380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.221359 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpbm\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-kube-api-access-7xpbm\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.221690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.221662 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5123781-6eea-46f8-8893-175421c25141-istiod-ca-cert\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.222509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.222488 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-data\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.222996 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.222973 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-credential-socket\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.257356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.257321 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-socket\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.274266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.261578 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:06.274565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.274525 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:06.291832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.261889 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-certs\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.315280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.315238 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:06.369966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.369929 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5123781-6eea-46f8-8893-175421c25141-istiod-ca-cert\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.528508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.370663 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-envoy\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.528772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.384756 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-istio-token\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.528892 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.457965 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:06.533142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.529199 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:32:06.543955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.463505 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:32:06.543955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.480089 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5123781-6eea-46f8-8893-175421c25141-istio-podinfo\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.543955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.490281 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpbm\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-kube-api-access-7xpbm\") pod \"e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.543955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.463473 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:06.543955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.542107 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:32:06.619543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.619508 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:06.767157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.763803 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt"] Nov 04 01:32:07.004204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:06.994191 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6"] Nov 04 01:32:07.060144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.059637 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:07.126288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.126236 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:32:07.140193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.133447 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz"] Nov 04 01:32:07.150470 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.150425 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.273203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.269578 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrrg\" (UniqueName: \"kubernetes.io/projected/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-kube-api-access-5hrrg\") pod \"payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6\" (UID: \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:07.273203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.270038 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6\" (UID: \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:07.313301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.308769 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:07.329320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.329270 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.477243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.442260 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.548810 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.535515 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.550421 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.550451 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.674445 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6\" (UID: \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.949726 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrrg\" (UniqueName: \"kubernetes.io/projected/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-kube-api-access-5hrrg\") pod \"payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6\" (UID: \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.949790 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz\" (UID: \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.949905 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j2ks\" (UniqueName: \"kubernetes.io/projected/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-kube-api-access-7j2ks\") pod \"payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz\" (UID: \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.752950 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.950196 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.684977 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.732389 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c75ea1-dockercfg-k7rcm\"" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.733648 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c75ea1\"" Nov 04 01:32:07.995205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:07.753679 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6\" (UID: \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:08.245291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.245197 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:32:08.336575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.276160 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2ks\" (UniqueName: \"kubernetes.io/projected/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-kube-api-access-7j2ks\") pod \"payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz\" (UID: \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:08.337698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.337663 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz\" (UID: \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:08.338089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.336442 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:32:08.338629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.276585 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Nov 04 01:32:08.350288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.350258 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:32:08.350405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.350375 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz"] Nov 04 01:32:08.350483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.350437 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6"] Nov 04 01:32:08.351761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.351727 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz\" (UID: \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:08.708352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.669500 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:08.708352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.695314 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:32:08.868156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.844319 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:08.970213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.945152 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrrg\" (UniqueName: \"kubernetes.io/projected/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-kube-api-access-5hrrg\") pod \"payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6\" (UID: \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:08.970559 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.970523 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:09.004146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.978746 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:32:09.004146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.989947 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:32:09.004146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:08.991850 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:32:09.173824 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.104409 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:32:09.226914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.199000 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:32:09.228167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.228103 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:09.228347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.228320 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:09.341074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.340745 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:09.355713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.350419 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2ks\" (UniqueName: \"kubernetes.io/projected/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-kube-api-access-7j2ks\") pod \"payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz\" (UID: \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\") " pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:09.528241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.449385 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:32:09.528241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.482562 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 04 01:32:09.648145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.635674 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:32:09.672142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.665385 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:09.908356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.875818 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:32:10.141184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.916448 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:10.141184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:09.981217 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:10.141184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:10.023270 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:10.141184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:10.023430 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:32:10.219511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:10.218888 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:10.219511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:10.218974 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:10.304166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:10.296701 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:32:10.533200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:10.455243 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:10.999145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:10.976267 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:11.077233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.033731 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:11.104145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.077471 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:11.124292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.095853 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" podUID="f80b3f88-24db-4e50-aecc-c1c19664e0ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Nov 04 01:32:11.370845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.366477 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:11.370845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.101196 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:11.370845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.366898 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:11.370845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.366958 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:11.370845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.367578 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:32:11.370845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.367895 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:32:11.412640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.412350 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:32:11.412801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.412744 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:32:11.413215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.043480 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:32:11.469011 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.468953 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:11.498259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.469481 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:11.498553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.498514 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:11.606163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.584200 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:11.618348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.618298 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:11.619068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.619034 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:11.780145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:11.763493 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Nov 04 01:32:12.002430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.002391 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:12.055374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.055263 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:12.131152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.126289 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded" Nov 04 01:32:12.223174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.223113 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:12.289148 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:12.265542 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.064s" Nov 04 01:32:12.403312 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.403222 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:32:12.403974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.403917 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:32:12.407157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.404245 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:32:12.407157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.406233 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:12.407157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.406272 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:12.484551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.469947 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:32:12.506305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.506249 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:12.517924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.517882 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:12.586527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:12.586482 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:13.156957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.156869 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:13.176165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.162325 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:13.422494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.407782 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:32:13.422494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.408185 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:32:13.496364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.494305 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:32:13.552146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.551269 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:13.552146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.551802 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:13.761195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.725216 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:13.834656 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:13.834617 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.569s" Nov 04 01:32:13.834991 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:13.834972 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:32:14.233964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.211088 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:14.233964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.211168 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:14.383147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.377201 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:32:14.412795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.411726 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:14.412795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.412141 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:32:14.430881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.430837 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt"] Nov 04 01:32:14.456144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.452055 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" event={"ID":"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec","Type":"ContainerStarted","Data":"f6f340e6bf1bd99ae817fd74e48a0d044c4b6a2002dfe2820884864d74a5c6a7"} Nov 04 01:32:14.498081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.497277 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:14.807272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.799668 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6"] Nov 04 01:32:14.807272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.804304 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:32:14.859706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:14.849662 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:32:15.193145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.191385 5051 patch_prober.go:29] interesting pod/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 04 01:32:15.193145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.191445 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 04 01:32:15.278724 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.263974 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:32:15.393349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.393292 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:32:15.434016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.433954 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:15.440151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.438034 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:15.623401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.623283 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:32:15.626709 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.626668 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" event={"ID":"c5123781-6eea-46f8-8893-175421c25141","Type":"ContainerStarted","Data":"5f690188e4f67915a00e38eec1ece7662a2eb39e00ba9e9d5d63ca7637a40f6e"} Nov 04 01:32:15.707160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.697540 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:15.731610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.731578 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:32:15.731893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.731872 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt"] Nov 04 01:32:15.732105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.732092 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz"] Nov 04 01:32:15.732407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.732394 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:15.732607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.732595 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:32:15.733430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.733402 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" containerID="cri-o://e3681e8a4a363778fcc523446ab34569cd3a3562430e97cbadbcc8947f47dc88" gracePeriod=30 Nov 04 01:32:15.840146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.807575 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:32:15.843278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.843212 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podStartSLOduration=11.843190642 podStartE2EDuration="11.843190642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:15.38861639 +0000 UTC m=+2074.360365263" watchObservedRunningTime="2026-11-04 01:32:15.843190642 +0000 UTC m=+2074.814939516" Nov 04 01:32:15.941216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:15.858743 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:32:16.065510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.065468 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:32:16.164145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.152277 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:32:16.395398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.395352 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:16.452396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.452353 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:16.623108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.623070 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" event={"ID":"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec","Type":"ContainerStarted","Data":"bd03f89f7b5f51402d35411305b58a7e0e7fdad71a0bc6e483eaa17bd7a25b43"} Nov 04 01:32:16.624421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.624395 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:16.624647 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.624622 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:16.631836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.631795 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" event={"ID":"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44","Type":"ContainerStarted","Data":"2adf1bb8b8d4041be74e5fa046cf4359e644ce77ee923fa4a762f56e09f14cd0"} Nov 04 01:32:16.633648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.633610 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:16.633862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.633834 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" event={"ID":"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44","Type":"ContainerStarted","Data":"0c8d4eac04378cfa971f8af3f2a3883e04cec3b8058953253aab3354990e86f8"} Nov 04 01:32:16.640184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.640159 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" event={"ID":"c5123781-6eea-46f8-8893-175421c25141","Type":"ContainerStarted","Data":"12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb"} Nov 04 01:32:16.640484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.640344 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:16.642947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.642929 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:32:16.811208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.811172 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:32:16.834657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:16.834626 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:32:17.040563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:17.040527 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:32:17.044909 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:17.044885 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:32:17.105257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:17.097754 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" podStartSLOduration=13.09773279 podStartE2EDuration="13.09773279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:16.780733831 +0000 UTC m=+2075.752482703" watchObservedRunningTime="2026-11-04 01:32:17.09773279 +0000 UTC m=+2076.069481662" Nov 04 01:32:17.163725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:17.152369 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:17.163930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:17.163216 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:17.327211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:17.326544 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" podStartSLOduration=13.326530533 podStartE2EDuration="13.326530533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:16.862583524 +0000 UTC m=+2075.834332395" watchObservedRunningTime="2026-11-04 01:32:17.326530533 +0000 UTC m=+2076.298279406" Nov 04 01:32:17.332551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:17.332524 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:32:17.332861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:17.332845 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:32:18.113983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.113944 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr"] Nov 04 01:32:18.122445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.121940 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.137783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.137758 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3888c434-openshift-default-dockercfg-lbhwx\"" Nov 04 01:32:18.144632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.144601 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q"] Nov 04 01:32:18.152320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.152297 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.256969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.256934 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507aa1e3-d0e1-4c14-ba76-3f271e737658-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.257261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.257225 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6b847-b5dc-45a5-b0b9-1859be473668-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.257480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.257448 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdw4\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-kube-api-access-dmdw4\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.257767 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.257732 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.257970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.257938 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.258151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.258115 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.258361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.258344 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.258517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.258501 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.258698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.258669 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.258893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.258875 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.259082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.259066 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.259266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.259234 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.266734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.266708 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.266933 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.266914 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmlgm\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-kube-api-access-wmlgm\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.267077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.267055 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.267358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.267324 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.462401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.462303 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.475314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.475272 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.563914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.563880 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr"] Nov 04 01:32:18.577695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.577661 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q"] Nov 04 01:32:18.597155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.592085 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.597417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.597396 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.597614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.597598 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.597745 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.597731 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.598048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.598027 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.598243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.598226 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.598408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.598393 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.598560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.598545 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.598762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.598746 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.598897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.598879 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmlgm\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-kube-api-access-wmlgm\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.599006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.598993 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.599228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.599211 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.599446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.599427 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.599568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.599553 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.603743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.603716 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507aa1e3-d0e1-4c14-ba76-3f271e737658-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.603934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.603916 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6b847-b5dc-45a5-b0b9-1859be473668-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.604795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.604052 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdw4\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-kube-api-access-dmdw4\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.604795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.604146 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.604795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.602644 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.604795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.602896 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.604795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.599964 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.605109 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.605015 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507aa1e3-d0e1-4c14-ba76-3f271e737658-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.605109 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.593809 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.605784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.605757 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6b847-b5dc-45a5-b0b9-1859be473668-istiod-ca-cert\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.605873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.600242 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.606076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.602057 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-data\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.606659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.606615 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-credential-socket\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.609727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.609697 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-certs\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.614285 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.614236 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.618593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.618563 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.619883 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.619848 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdw4\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-kube-api-access-dmdw4\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.623598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.623562 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmlgm\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-kube-api-access-wmlgm\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.625758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.625678 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-token\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.626238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.626017 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.626238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.626195 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-envoy\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.626517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.626442 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-podinfo\") pod \"e2e-mt-3888c434-openshift-default-75b6d654bf-v475q\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:18.759169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.757456 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:18.900381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:18.897657 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:19.159619 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:32:19.159584 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d6b847_b5dc_45a5_b0b9_1859be473668.slice/crio-b5e51e48b5fe60cc3eba946cdcb490ea5f95ad0f55bb3dddf06196775f9b147c WatchSource:0}: Error finding container b5e51e48b5fe60cc3eba946cdcb490ea5f95ad0f55bb3dddf06196775f9b147c: Status 404 returned error can't find the container with id b5e51e48b5fe60cc3eba946cdcb490ea5f95ad0f55bb3dddf06196775f9b147c Nov 04 01:32:19.162051 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.161984 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr"] Nov 04 01:32:19.163352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.163319 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:19.163417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.163386 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:19.163460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.163416 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:19.248911 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:32:19.248865 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507aa1e3_d0e1_4c14_ba76_3f271e737658.slice/crio-eb40088e124d006dc4a402b7fc484de95d12a98c0dae69c5bbeae4c648e25535 WatchSource:0}: Error finding container eb40088e124d006dc4a402b7fc484de95d12a98c0dae69c5bbeae4c648e25535: Status 404 returned error can't find the container with id eb40088e124d006dc4a402b7fc484de95d12a98c0dae69c5bbeae4c648e25535 Nov 04 01:32:19.250627 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.250602 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q"] Nov 04 01:32:19.252564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.252535 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:19.252643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.252624 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:19.252685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.252673 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:19.872143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.869515 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" event={"ID":"61d6b847-b5dc-45a5-b0b9-1859be473668","Type":"ContainerStarted","Data":"8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838"} Nov 04 01:32:19.872143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.869582 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" event={"ID":"61d6b847-b5dc-45a5-b0b9-1859be473668","Type":"ContainerStarted","Data":"b5e51e48b5fe60cc3eba946cdcb490ea5f95ad0f55bb3dddf06196775f9b147c"} Nov 04 01:32:19.948695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.940540 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" event={"ID":"507aa1e3-d0e1-4c14-ba76-3f271e737658","Type":"ContainerStarted","Data":"24288ca1a8c371202d4ee61332ffeba5299b41de612895061b5de9700c5eb36c"} Nov 04 01:32:19.948695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.940611 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" event={"ID":"507aa1e3-d0e1-4c14-ba76-3f271e737658","Type":"ContainerStarted","Data":"eb40088e124d006dc4a402b7fc484de95d12a98c0dae69c5bbeae4c648e25535"} Nov 04 01:32:19.981486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:19.981427 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podStartSLOduration=1.9814064120000001 podStartE2EDuration="1.981406412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:19.932806996 +0000 UTC m=+2078.904555880" watchObservedRunningTime="2026-11-04 01:32:19.981406412 +0000 UTC m=+2078.953155284" Nov 04 01:32:20.086978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:20.078572 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podStartSLOduration=2.078550242 podStartE2EDuration="2.078550242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:20.013937726 +0000 UTC m=+2078.985686598" watchObservedRunningTime="2026-11-04 01:32:20.078550242 +0000 UTC m=+2079.050299114" Nov 04 01:32:20.812179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:20.765178 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:20.812179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:20.767522 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:20.812179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:20.767615 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:20.921153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:20.908669 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:20.921153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:20.908877 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:20.921153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:20.908931 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:21.244954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:21.244517 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2"] Nov 04 01:32:21.244954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:21.244725 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g"] Nov 04 01:32:21.770185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:21.767376 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:21.770185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:21.767434 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:21.903143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:21.899036 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:21.903143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:21.899099 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:22.160145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.141928 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr"] Nov 04 01:32:22.204843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.204810 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.241327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.240842 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.241327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.240935 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.241327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.240971 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.241327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.241030 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d27dcf-aac3-4773-bc85-20aed144185d-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.241327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.241100 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.241327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.241223 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.242344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.242031 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpnsr\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-kube-api-access-wpnsr\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.242882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.242562 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.242882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.242606 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.325273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.325193 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr"] Nov 04 01:32:22.345029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.344995 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.345199 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.345157 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.345268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.345235 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.345335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.345268 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.345335 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.345329 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d27dcf-aac3-4773-bc85-20aed144185d-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.345440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.345395 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.345518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.345499 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.345601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.345586 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpnsr\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-kube-api-access-wpnsr\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.345712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.345698 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.346175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.346156 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-credential-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.346441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.346413 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-socket\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.347171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.346647 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-certs\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.348495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.347458 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-data\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.348495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.348111 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d27dcf-aac3-4773-bc85-20aed144185d-istiod-ca-cert\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.382396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.375999 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-podinfo\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.395871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.395797 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-envoy\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.440369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.439924 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-token\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.440369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.440310 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpnsr\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-kube-api-access-wpnsr\") pod \"e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.579252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.538140 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:22.782207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.778498 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:22.782207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.778556 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:22.929266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.920699 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:22.929266 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:22.920754 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:23.207735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.176314 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" event={"ID":"73872956-f8af-4631-b5e3-a88f3d39f3d9","Type":"ContainerDied","Data":"e3681e8a4a363778fcc523446ab34569cd3a3562430e97cbadbcc8947f47dc88"} Nov 04 01:32:23.207735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.176443 5051 generic.go:356] "Generic (PLEG): container finished" podID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerID="e3681e8a4a363778fcc523446ab34569cd3a3562430e97cbadbcc8947f47dc88" exitCode=0 Nov 04 01:32:23.785148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.768311 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr"] Nov 04 01:32:23.929695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.816345 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:23.931575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.930109 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:23.931803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.922932 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:23.932009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.931982 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:23.934832 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:32:23.934622 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d27dcf_aac3_4773_bc85_20aed144185d.slice/crio-31fd3f216f27e5c27ee11aa4840e86ce17fd70807d6a029543e3fc9e5a658c12 WatchSource:0}: Error finding container 31fd3f216f27e5c27ee11aa4840e86ce17fd70807d6a029543e3fc9e5a658c12: Status 404 returned error can't find the container with id 31fd3f216f27e5c27ee11aa4840e86ce17fd70807d6a029543e3fc9e5a658c12 Nov 04 01:32:23.943159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.939965 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:23.943159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.942016 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:23.943159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.942091 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:23.943159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:23.942147 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:24.411857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.393320 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-podinfo\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.433622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.433535 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73872956-f8af-4631-b5e3-a88f3d39f3d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73872956-f8af-4631-b5e3-a88f3d39f3d9-istiod-ca-cert\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.434080 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.434056 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-data\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.434265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.434248 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-certs\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.449256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.449220 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-credential-socket\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.449531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.449511 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-kube-api-access-kbgng\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-kube-api-access-kbgng\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.449687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.449671 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-envoy\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.466396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.449882 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-socket\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.466797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.466757 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-token\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-token\") pod \"73872956-f8af-4631-b5e3-a88f3d39f3d9\" (UID: \"73872956-f8af-4631-b5e3-a88f3d39f3d9\") " Nov 04 01:32:24.604357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.542888 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:24.606784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.606252 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" event={"ID":"b1d27dcf-aac3-4773-bc85-20aed144185d","Type":"ContainerStarted","Data":"31fd3f216f27e5c27ee11aa4840e86ce17fd70807d6a029543e3fc9e5a658c12"} Nov 04 01:32:24.695546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.445233 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-certs" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:24.695763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.535414 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-data" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:24.746247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.535460 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-credential-socket" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:24.746451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.546626 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73872956-f8af-4631-b5e3-a88f3d39f3d9-istiod-ca-cert" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:24.746548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.645448 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-socket" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:24.746658 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.679245 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" event={"ID":"73872956-f8af-4631-b5e3-a88f3d39f3d9","Type":"ContainerDied","Data":"7c30b1ffd201aed341ab50902791f26a7575567a4a6d9a985bfda6d0fb7783c6"} Nov 04 01:32:24.746882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.746861 5051 scope.go:122] "RemoveContainer" containerID="e3681e8a4a363778fcc523446ab34569cd3a3562430e97cbadbcc8947f47dc88" Nov 04 01:32:24.770645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.769848 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-kube-api-access-kbgng" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "kube-api-access-kbgng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:24.771882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.771382 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:24.771882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.771443 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:24.780994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.780531 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:24.780994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.780561 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:24.780994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.780577 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbgng\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-kube-api-access-kbgng\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:24.780994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.780592 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:24.780994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.780606 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73872956-f8af-4631-b5e3-a88f3d39f3d9-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:24.780994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.780618 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:24.793455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.793428 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt" Nov 04 01:32:24.877389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.877351 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-podinfo" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:24.908145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.902968 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:24.908145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:24.903106 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:25.007296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.007247 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:32:25.029060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.011104 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-token" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:25.029060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.012109 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:25.029060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.012154 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:25.093153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.077257 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-envoy" pod "73872956-f8af-4631-b5e3-a88f3d39f3d9" (UID: "73872956-f8af-4631-b5e3-a88f3d39f3d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:25.251685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.183525 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73872956-f8af-4631-b5e3-a88f3d39f3d9-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:25.355500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.355457 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:25.371162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.356656 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:32:25.407551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.405953 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:32:25.413349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.413316 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:25.448990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.448412 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:25.448990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.448501 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" podUID="3365fd0a-a069-4da8-bd68-b323fb2411ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:25.545748 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.539255 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:25.777363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.777323 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:25.777657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.777624 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:25.974198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.952226 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:25.980154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:25.979249 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:26.263098 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:26.254260 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt"] Nov 04 01:32:26.353145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:26.315186 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" podUID="7527ccd3-d55b-45b8-9fe7-66553840fd76" containerName="payload-processing" containerID="cri-o://4eaa1a0909e62ae88ba61a0b467cc91e98e292c326ecc494a562b2ce53aab8dc" gracePeriod=25 Nov 04 01:32:26.818149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:26.818058 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:26.818149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:26.818133 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:26.947163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:26.927269 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:26.947163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:26.927347 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:27.013997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.013948 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:27.043262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.043226 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-30b1a1-openshift-default-55565689d5-t27qt"] Nov 04 01:32:27.138020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.082518 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" event={"ID":"b1d27dcf-aac3-4773-bc85-20aed144185d","Type":"ContainerStarted","Data":"661ba69a12888e72adb44d541f75b6598392ac83eb28f1909712a1682f70cba2"} Nov 04 01:32:27.275188 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.269570 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:27.347095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.340240 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" podUID="4b2256e0-d640-44f4-937e-a4f3570b2211" containerName="payload-pre-processing" containerID="cri-o://8a3cf3a8859e00d6c8fdf76803e09008c24f397272f8a56a087eac237a47682f" gracePeriod=25 Nov 04 01:32:27.544953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.536713 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:32:27.808670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.767916 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:32:27.808670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.798029 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:27.808670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:27.798103 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:28.155764 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:28.154414 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73872956_f8af_4631_b5e3_a88f3d39f3d9.slice/crio-7c30b1ffd201aed341ab50902791f26a7575567a4a6d9a985bfda6d0fb7783c6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7527ccd3_d55b_45b8_9fe7_66553840fd76.slice/crio-conmon-4eaa1a0909e62ae88ba61a0b467cc91e98e292c326ecc494a562b2ce53aab8dc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73872956_f8af_4631_b5e3_a88f3d39f3d9.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:32:28.171152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.170273 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podStartSLOduration=6.170253884 podStartE2EDuration="6.170253884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:27.49004517 +0000 UTC m=+2086.461794041" watchObservedRunningTime="2026-11-04 01:32:28.170253884 +0000 UTC m=+2087.142002755" Nov 04 01:32:28.191606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.191379 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:28.191606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.191436 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:28.191606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.191508 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:28.191606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.191539 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:28.418852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.340597 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:28.547476 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.546670 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:28.566547 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.548967 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:32:28.568687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.546317 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:28.601657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.576617 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" path="/var/lib/kubelet/pods/73872956-f8af-4631-b5e3-a88f3d39f3d9/volumes" Nov 04 01:32:28.601657 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:28.577224 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.335s" Nov 04 01:32:28.601657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.577263 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:28.601657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.577352 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:32:28.601657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.577371 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2"] Nov 04 01:32:28.601657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.577408 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486"] Nov 04 01:32:28.601657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.577462 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:32:28.601657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.577501 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:32:28.609943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.609912 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:32:28.624149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.612641 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:28.624149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.612702 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:28.630062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.625551 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:28.777924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.774681 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:28.777924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.774742 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:28.906136 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.906082 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:28.906432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.906403 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:28.922718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.922684 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:32:28.970793 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:28.925491 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2256e0_d640_44f4_937e_a4f3570b2211.slice/crio-conmon-8a3cf3a8859e00d6c8fdf76803e09008c24f397272f8a56a087eac237a47682f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2256e0_d640_44f4_937e_a4f3570b2211.slice/crio-8a3cf3a8859e00d6c8fdf76803e09008c24f397272f8a56a087eac237a47682f.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:32:28.970793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:28.938378 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:32:29.139471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.132178 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:29.139471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.135032 5051 generic.go:356] "Generic (PLEG): container finished" podID="7527ccd3-d55b-45b8-9fe7-66553840fd76" containerID="4eaa1a0909e62ae88ba61a0b467cc91e98e292c326ecc494a562b2ce53aab8dc" exitCode=0 Nov 04 01:32:29.160626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.156849 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" event={"ID":"7527ccd3-d55b-45b8-9fe7-66553840fd76","Type":"ContainerDied","Data":"4eaa1a0909e62ae88ba61a0b467cc91e98e292c326ecc494a562b2ce53aab8dc"} Nov 04 01:32:29.188790 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.178304 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:29.301942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.282950 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:29.353864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.351391 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:32:29.354245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.354215 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:32:29.455906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.445906 5051 generic.go:356] "Generic (PLEG): container finished" podID="4b2256e0-d640-44f4-937e-a4f3570b2211" containerID="8a3cf3a8859e00d6c8fdf76803e09008c24f397272f8a56a087eac237a47682f" exitCode=0 Nov 04 01:32:29.573085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.573041 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:29.584487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.574281 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:29.629568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.629518 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:29.729766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.721648 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7527ccd3-d55b-45b8-9fe7-66553840fd76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7527ccd3-d55b-45b8-9fe7-66553840fd76-plugins-config-volume\") pod \"7527ccd3-d55b-45b8-9fe7-66553840fd76\" (UID: \"7527ccd3-d55b-45b8-9fe7-66553840fd76\") " Nov 04 01:32:29.729766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.721746 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7527ccd3-d55b-45b8-9fe7-66553840fd76-kube-api-access-dzzpr\" (UniqueName: \"kubernetes.io/projected/7527ccd3-d55b-45b8-9fe7-66553840fd76-kube-api-access-dzzpr\") pod \"7527ccd3-d55b-45b8-9fe7-66553840fd76\" (UID: \"7527ccd3-d55b-45b8-9fe7-66553840fd76\") " Nov 04 01:32:29.753228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.753185 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7527ccd3-d55b-45b8-9fe7-66553840fd76-plugins-config-volume" pod "7527ccd3-d55b-45b8-9fe7-66553840fd76" (UID: "7527ccd3-d55b-45b8-9fe7-66553840fd76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:29.754055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.754029 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7527ccd3-d55b-45b8-9fe7-66553840fd76-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.776844 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.776799 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:32:29.777038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.776878 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:32:29.777038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.776909 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:32:29.777038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.776937 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:32:29.777038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.776960 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" event={"ID":"4b2256e0-d640-44f4-937e-a4f3570b2211","Type":"ContainerDied","Data":"8a3cf3a8859e00d6c8fdf76803e09008c24f397272f8a56a087eac237a47682f"} Nov 04 01:32:29.811934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.811890 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 04 01:32:29.822749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.822713 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 04 01:32:29.833569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.833497 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7527ccd3-d55b-45b8-9fe7-66553840fd76-kube-api-access-dzzpr" pod "7527ccd3-d55b-45b8-9fe7-66553840fd76" (UID: "7527ccd3-d55b-45b8-9fe7-66553840fd76"). InnerVolumeSpecName "kube-api-access-dzzpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:29.909444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.909405 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzzpr\" (UniqueName: \"kubernetes.io/projected/7527ccd3-d55b-45b8-9fe7-66553840fd76-kube-api-access-dzzpr\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:29.913061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.913031 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 04 01:32:29.913321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:29.913296 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 04 01:32:30.323185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.322732 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:32:30.517254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.496415 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" event={"ID":"7527ccd3-d55b-45b8-9fe7-66553840fd76","Type":"ContainerDied","Data":"3d8ec3345633d591334ae92dafb497f0c6c5f475342b02db599fb67ab83f6200"} Nov 04 01:32:30.517254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.496489 5051 scope.go:122] "RemoveContainer" containerID="4eaa1a0909e62ae88ba61a0b467cc91e98e292c326ecc494a562b2ce53aab8dc" Nov 04 01:32:30.517254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.496632 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2" Nov 04 01:32:30.543536 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.543499 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" Nov 04 01:32:30.543760 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.543723 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" event={"ID":"4b2256e0-d640-44f4-937e-a4f3570b2211","Type":"ContainerDied","Data":"6513d5ab0bcdcc7c041ac74e2e6708e889e671b578f45388912ae91bdf97a9a1"} Nov 04 01:32:30.552269 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.551564 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2256e0-d640-44f4-937e-a4f3570b2211-kube-api-access-q276q\" (UniqueName: \"kubernetes.io/projected/4b2256e0-d640-44f4-937e-a4f3570b2211-kube-api-access-q276q\") pod \"4b2256e0-d640-44f4-937e-a4f3570b2211\" (UID: \"4b2256e0-d640-44f4-937e-a4f3570b2211\") " Nov 04 01:32:30.552269 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.551806 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2256e0-d640-44f4-937e-a4f3570b2211-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2256e0-d640-44f4-937e-a4f3570b2211-plugins-config-volume\") pod \"4b2256e0-d640-44f4-937e-a4f3570b2211\" (UID: \"4b2256e0-d640-44f4-937e-a4f3570b2211\") " Nov 04 01:32:30.560792 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.560764 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2256e0-d640-44f4-937e-a4f3570b2211-plugins-config-volume" pod "4b2256e0-d640-44f4-937e-a4f3570b2211" (UID: "4b2256e0-d640-44f4-937e-a4f3570b2211"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:30.562413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.562391 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:30.562631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.562607 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:30.588143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.579321 5051 scope.go:122] "RemoveContainer" containerID="8a3cf3a8859e00d6c8fdf76803e09008c24f397272f8a56a087eac237a47682f" Nov 04 01:32:30.591792 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.579313 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g" podUID="4b2256e0-d640-44f4-937e-a4f3570b2211" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.188:50052\" within 1s: context deadline exceeded" Nov 04 01:32:30.786978 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:32:30.579481 5051 logging.go:55] [core] [Channel #867 SubChannel #868]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.188:50052", ServerName: "10.132.0.188:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.188:50052: operation was canceled" Nov 04 01:32:30.827816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.802032 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2256e0-d640-44f4-937e-a4f3570b2211-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:30.989899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.944171 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2"] Nov 04 01:32:30.989899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.944226 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-50dab0-cd4f5d64d-vhtf2"] Nov 04 01:32:30.989899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:30.945754 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2256e0-d640-44f4-937e-a4f3570b2211-kube-api-access-q276q" pod "4b2256e0-d640-44f4-937e-a4f3570b2211" (UID: "4b2256e0-d640-44f4-937e-a4f3570b2211"). InnerVolumeSpecName "kube-api-access-q276q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:31.025197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.008002 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q276q\" (UniqueName: \"kubernetes.io/projected/4b2256e0-d640-44f4-937e-a4f3570b2211-kube-api-access-q276q\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:31.144424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.144337 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc"] Nov 04 01:32:31.145705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.145670 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7527ccd3-d55b-45b8-9fe7-66553840fd76" containerName="payload-processing" Nov 04 01:32:31.145705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.145702 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7527ccd3-d55b-45b8-9fe7-66553840fd76" containerName="payload-processing" Nov 04 01:32:31.145935 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.145750 5051 cpu_manager.go:414] "containerMap: removing container" podUID="4b2256e0-d640-44f4-937e-a4f3570b2211" containerName="payload-pre-processing" Nov 04 01:32:31.145935 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.145760 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2256e0-d640-44f4-937e-a4f3570b2211" containerName="payload-pre-processing" Nov 04 01:32:31.145935 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.145808 5051 cpu_manager.go:414] "containerMap: removing container" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" Nov 04 01:32:31.145935 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.145817 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" Nov 04 01:32:31.146234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.146208 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="73872956-f8af-4631-b5e3-a88f3d39f3d9" containerName="istio-proxy" Nov 04 01:32:31.146319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.146276 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7527ccd3-d55b-45b8-9fe7-66553840fd76" containerName="payload-processing" Nov 04 01:32:31.146319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.146295 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2256e0-d640-44f4-937e-a4f3570b2211" containerName="payload-pre-processing" Nov 04 01:32:31.555189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.551205 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc"] Nov 04 01:32:31.555189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.551362 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg"] Nov 04 01:32:31.555189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.551466 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:31.614561 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.614525 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8e9791\"" Nov 04 01:32:31.658706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.642442 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg"] Nov 04 01:32:31.806196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.779821 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:31.810091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.756537 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352949e5-7e2e-45da-87ab-7350a071772a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc\" (UID: \"352949e5-7e2e-45da-87ab-7350a071772a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:31.810471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.810447 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmk7b\" (UniqueName: \"kubernetes.io/projected/352949e5-7e2e-45da-87ab-7350a071772a-kube-api-access-dmk7b\") pod \"payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc\" (UID: \"352949e5-7e2e-45da-87ab-7350a071772a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:31.819147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.810780 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg\" (UID: \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:31.830251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.819912 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-kube-api-access-9tww6\") pod \"payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg\" (UID: \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:31.856145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.653680 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:31.874143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.810058 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:31.874143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.856649 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:31.874143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.869091 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:31.951178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.895553 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8e9791-dockercfg-ntdsg\"" Nov 04 01:32:31.951178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.945438 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352949e5-7e2e-45da-87ab-7350a071772a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc\" (UID: \"352949e5-7e2e-45da-87ab-7350a071772a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:31.951178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.945565 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmk7b\" (UniqueName: \"kubernetes.io/projected/352949e5-7e2e-45da-87ab-7350a071772a-kube-api-access-dmk7b\") pod \"payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc\" (UID: \"352949e5-7e2e-45da-87ab-7350a071772a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:31.951178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.945745 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg\" (UID: \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:31.951178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.945870 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-kube-api-access-9tww6\") pod \"payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg\" (UID: \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:31.951178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.947104 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352949e5-7e2e-45da-87ab-7350a071772a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc\" (UID: \"352949e5-7e2e-45da-87ab-7350a071772a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:31.969976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.965749 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:31.993255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:31.969918 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:32.040492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.040454 5051 patch_prober.go:29] interesting pod/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:32.040851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.040816 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:32.074967 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.074881 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg\" (UID: \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:32.086143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.080853 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7527ccd3-d55b-45b8-9fe7-66553840fd76" path="/var/lib/kubelet/pods/7527ccd3-d55b-45b8-9fe7-66553840fd76/volumes" Nov 04 01:32:32.098046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.087001 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g"] Nov 04 01:32:32.124215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.112107 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-50dab0-67d785f6c5-vpv9g"] Nov 04 01:32:32.162801 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.162657 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-kube-api-access-9tww6\") pod \"payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg\" (UID: \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\") " pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:32.287868 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.273909 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:32.613322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.580995 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:32.613322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.581058 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:32.635884 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.632423 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmk7b\" (UniqueName: \"kubernetes.io/projected/352949e5-7e2e-45da-87ab-7350a071772a-kube-api-access-dmk7b\") pod \"payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc\" (UID: \"352949e5-7e2e-45da-87ab-7350a071772a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:32.845392 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.843380 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:32.867895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.867801 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:32.900230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.889405 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:32.963542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:32.963509 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:33.038776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.038743 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:33.038929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.038822 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:33.069644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.063379 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:33.100358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.100324 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:33.115571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.105930 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:33.255723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.252348 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2256e0-d640-44f4-937e-a4f3570b2211" path="/var/lib/kubelet/pods/4b2256e0-d640-44f4-937e-a4f3570b2211/volumes" Nov 04 01:32:33.393955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.393871 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg"] Nov 04 01:32:33.572640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.572606 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:33.573143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.573011 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:33.610312 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.610234 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" podUID="5008b044-5916-4244-9e18-a1478be5e552" containerName="payload-pre-processing" containerID="cri-o://f806114c71ad76dcee76b0f8e8eed71119456f326833fe309a6297247f938955" gracePeriod=25 Nov 04 01:32:33.610731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.610701 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" podUID="a782226f-9d6a-4948-979d-d8efc336678a" containerName="payload-processing" containerID="cri-o://c98a67b442828e74f5cb9261328bfde98532daed193bbc10e7d63de45715f0ec" gracePeriod=25 Nov 04 01:32:33.985503 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:32:33.985449 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352949e5_7e2e_45da_87ab_7350a071772a.slice/crio-de4e1a49e0f216a94fbada218ba16b31a8f3944fcf0a6a5a8a81f00ab126edc4 WatchSource:0}: Error finding container de4e1a49e0f216a94fbada218ba16b31a8f3944fcf0a6a5a8a81f00ab126edc4: Status 404 returned error can't find the container with id de4e1a49e0f216a94fbada218ba16b31a8f3944fcf0a6a5a8a81f00ab126edc4 Nov 04 01:32:33.996726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:33.996689 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc"] Nov 04 01:32:34.145785 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:34.144395 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:32:34.184142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.176752 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" event={"ID":"8193ef32-0b21-49a2-9494-5d6ad52cdcf2","Type":"ContainerStarted","Data":"a93a45fdeae66518c5e8d84b75c0ac9ca35d113bc9f2b5452d6facaa26462395"} Nov 04 01:32:34.191181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.191110 5051 generic.go:356] "Generic (PLEG): container finished" podID="a782226f-9d6a-4948-979d-d8efc336678a" containerID="c98a67b442828e74f5cb9261328bfde98532daed193bbc10e7d63de45715f0ec" exitCode=0 Nov 04 01:32:34.191442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.191420 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" event={"ID":"a782226f-9d6a-4948-979d-d8efc336678a","Type":"ContainerDied","Data":"c98a67b442828e74f5cb9261328bfde98532daed193bbc10e7d63de45715f0ec"} Nov 04 01:32:34.193600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.193574 5051 generic.go:356] "Generic (PLEG): container finished" podID="5008b044-5916-4244-9e18-a1478be5e552" containerID="f806114c71ad76dcee76b0f8e8eed71119456f326833fe309a6297247f938955" exitCode=0 Nov 04 01:32:34.193920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.193899 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" event={"ID":"5008b044-5916-4244-9e18-a1478be5e552","Type":"ContainerDied","Data":"f806114c71ad76dcee76b0f8e8eed71119456f326833fe309a6297247f938955"} Nov 04 01:32:34.196730 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.196704 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" event={"ID":"352949e5-7e2e-45da-87ab-7350a071772a","Type":"ContainerStarted","Data":"de4e1a49e0f216a94fbada218ba16b31a8f3944fcf0a6a5a8a81f00ab126edc4"} Nov 04 01:32:34.608273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.607771 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:34.608273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.607839 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:34.908070 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:34.907703 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:32:35.019140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.018391 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a782226f-9d6a-4948-979d-d8efc336678a-kube-api-access-gt895\" (UniqueName: \"kubernetes.io/projected/a782226f-9d6a-4948-979d-d8efc336678a-kube-api-access-gt895\") pod \"a782226f-9d6a-4948-979d-d8efc336678a\" (UID: \"a782226f-9d6a-4948-979d-d8efc336678a\") " Nov 04 01:32:35.019140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.018748 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a782226f-9d6a-4948-979d-d8efc336678a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782226f-9d6a-4948-979d-d8efc336678a-plugins-config-volume\") pod \"a782226f-9d6a-4948-979d-d8efc336678a\" (UID: \"a782226f-9d6a-4948-979d-d8efc336678a\") " Nov 04 01:32:35.021427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.021356 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a782226f-9d6a-4948-979d-d8efc336678a-plugins-config-volume" pod "a782226f-9d6a-4948-979d-d8efc336678a" (UID: "a782226f-9d6a-4948-979d-d8efc336678a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:35.024477 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.024446 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a782226f-9d6a-4948-979d-d8efc336678a-kube-api-access-gt895" pod "a782226f-9d6a-4948-979d-d8efc336678a" (UID: "a782226f-9d6a-4948-979d-d8efc336678a"). InnerVolumeSpecName "kube-api-access-gt895". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:35.121140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.121082 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt895\" (UniqueName: \"kubernetes.io/projected/a782226f-9d6a-4948-979d-d8efc336678a-kube-api-access-gt895\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:35.121140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.121138 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a782226f-9d6a-4948-979d-d8efc336678a-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:35.199406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.199312 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:32:35.305779 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.299311 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" event={"ID":"352949e5-7e2e-45da-87ab-7350a071772a","Type":"ContainerStarted","Data":"516ae398957db60f1ad6069439ca451491dad585439b1793c2f6ebff5ce63d65"} Nov 04 01:32:35.305983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.305842 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:35.307468 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.307434 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" event={"ID":"8193ef32-0b21-49a2-9494-5d6ad52cdcf2","Type":"ContainerStarted","Data":"101da47d822f52c9fdca6b2707502cd13f9ea0a95feb76fe5f504ae3d94df75b"} Nov 04 01:32:35.307821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.307796 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:35.313092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.312783 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" event={"ID":"a782226f-9d6a-4948-979d-d8efc336678a","Type":"ContainerDied","Data":"dad37c4d312e7d5a3642ca4a6cfb782ef99063ba6e5b24abe8779471698b8766"} Nov 04 01:32:35.313092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.312838 5051 scope.go:122] "RemoveContainer" containerID="c98a67b442828e74f5cb9261328bfde98532daed193bbc10e7d63de45715f0ec" Nov 04 01:32:35.313092 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.312975 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2" Nov 04 01:32:35.330010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.329976 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5008b044-5916-4244-9e18-a1478be5e552-kube-api-access-fkm49\" (UniqueName: \"kubernetes.io/projected/5008b044-5916-4244-9e18-a1478be5e552-kube-api-access-fkm49\") pod \"5008b044-5916-4244-9e18-a1478be5e552\" (UID: \"5008b044-5916-4244-9e18-a1478be5e552\") " Nov 04 01:32:35.330203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.330180 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5008b044-5916-4244-9e18-a1478be5e552-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5008b044-5916-4244-9e18-a1478be5e552-plugins-config-volume\") pod \"5008b044-5916-4244-9e18-a1478be5e552\" (UID: \"5008b044-5916-4244-9e18-a1478be5e552\") " Nov 04 01:32:35.331913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.331865 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" podStartSLOduration=4.329999877 podStartE2EDuration="4.329999877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:35.321721658 +0000 UTC m=+2094.293470529" watchObservedRunningTime="2026-11-04 01:32:35.329999877 +0000 UTC m=+2094.301748749" Nov 04 01:32:35.332745 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.332707 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5008b044-5916-4244-9e18-a1478be5e552-plugins-config-volume" pod "5008b044-5916-4244-9e18-a1478be5e552" (UID: "5008b044-5916-4244-9e18-a1478be5e552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:35.333174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.332950 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" Nov 04 01:32:35.335420 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.335283 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486" event={"ID":"5008b044-5916-4244-9e18-a1478be5e552","Type":"ContainerDied","Data":"8ecd3b2bd50776363519cfe4a9355e3a514ee4a3683b04b0ba6d090d6c0c0764"} Nov 04 01:32:35.411770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.411368 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5008b044-5916-4244-9e18-a1478be5e552-kube-api-access-fkm49" pod "5008b044-5916-4244-9e18-a1478be5e552" (UID: "5008b044-5916-4244-9e18-a1478be5e552"). InnerVolumeSpecName "kube-api-access-fkm49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:35.433545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.433030 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" podStartSLOduration=4.43300879 podStartE2EDuration="4.43300879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:35.410613779 +0000 UTC m=+2094.382362675" watchObservedRunningTime="2026-11-04 01:32:35.43300879 +0000 UTC m=+2094.404757663" Nov 04 01:32:35.444956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.441710 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkm49\" (UniqueName: \"kubernetes.io/projected/5008b044-5916-4244-9e18-a1478be5e552-kube-api-access-fkm49\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:35.444956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.441748 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5008b044-5916-4244-9e18-a1478be5e552-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:35.582515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.516912 5051 scope.go:122] "RemoveContainer" containerID="f806114c71ad76dcee76b0f8e8eed71119456f326833fe309a6297247f938955" Nov 04 01:32:35.582515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.556501 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 04 01:32:35.582515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.556557 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 04 01:32:35.610380 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:35.610342 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2"] Nov 04 01:32:36.209281 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:36.203918 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3888c434-68c74b4b4b-9hqv2"] Nov 04 01:32:36.248640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:36.248606 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486"] Nov 04 01:32:36.259902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:36.259864 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3888c434-7966d949c9-kc486"] Nov 04 01:32:36.757771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:36.757724 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:36.759144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:36.759031 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:36.760330 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:36.760283 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:37.200714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:37.200623 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5008b044-5916-4244-9e18-a1478be5e552" path="/var/lib/kubelet/pods/5008b044-5916-4244-9e18-a1478be5e552/volumes" Nov 04 01:32:37.343196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:37.342525 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a782226f-9d6a-4948-979d-d8efc336678a" path="/var/lib/kubelet/pods/a782226f-9d6a-4948-979d-d8efc336678a/volumes" Nov 04 01:32:38.043108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:38.043063 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4"] Nov 04 01:32:38.051848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:38.051809 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46bef393-m22c4"] Nov 04 01:32:39.302156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:39.293459 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3447fade-e00b-48a5-823a-2ace4656486f" path="/var/lib/kubelet/pods/3447fade-e00b-48a5-823a-2ace4656486f/volumes" Nov 04 01:32:40.394749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:40.394327 5051 patch_prober.go:29] interesting pod/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:40.394749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:40.394407 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" podUID="3365fd0a-a069-4da8-bd68-b323fb2411ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:41.325351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:41.325311 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9"] Nov 04 01:32:41.325972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:41.325938 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" podUID="3365fd0a-a069-4da8-bd68-b323fb2411ca" containerName="istio-proxy" containerID="cri-o://7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79" gracePeriod=30 Nov 04 01:32:41.345636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:41.344897 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr"] Nov 04 01:32:41.345636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:41.345189 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" containerID="cri-o://661ba69a12888e72adb44d541f75b6598392ac83eb28f1909712a1682f70cba2" gracePeriod=30 Nov 04 01:32:42.466151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:42.461683 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:42.582970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:42.582923 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:32:42.585924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:42.585766 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:32:43.918539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:43.918504 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr"] Nov 04 01:32:43.919053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:43.918796 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" containerID="cri-o://8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838" gracePeriod=30 Nov 04 01:32:43.927709 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:43.927676 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q"] Nov 04 01:32:43.929399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:43.929370 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" containerID="cri-o://24288ca1a8c371202d4ee61332ffeba5299b41de612895061b5de9700c5eb36c" gracePeriod=30 Nov 04 01:32:46.430707 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:46.428526 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:32:46.436211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:46.431482 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:32:46.927988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:46.927943 5051 generic.go:356] "Generic (PLEG): container finished" podID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerID="661ba69a12888e72adb44d541f75b6598392ac83eb28f1909712a1682f70cba2" exitCode=0 Nov 04 01:32:46.928176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:46.927984 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" event={"ID":"b1d27dcf-aac3-4773-bc85-20aed144185d","Type":"ContainerDied","Data":"661ba69a12888e72adb44d541f75b6598392ac83eb28f1909712a1682f70cba2"} Nov 04 01:32:46.973160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:46.972093 5051 patch_prober.go:29] interesting pod/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:46.973160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:46.972203 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:47.416147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.381103 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:47.506004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.487950 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-data\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.506004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.488089 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-credential-socket\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.506004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.488144 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-token\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-token\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.506004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.488181 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-socket\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.506004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.488238 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d27dcf-aac3-4773-bc85-20aed144185d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d27dcf-aac3-4773-bc85-20aed144185d-istiod-ca-cert\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.506004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.488423 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-kube-api-access-wpnsr\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-kube-api-access-wpnsr\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.506004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.488599 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-podinfo\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.506004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.488696 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-envoy\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.519034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.513433 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-data" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.519034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.516432 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-socket" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.519034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.516763 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d27dcf-aac3-4773-bc85-20aed144185d-istiod-ca-cert" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:47.519034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.517060 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-credential-socket" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.519349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.519322 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-kube-api-access-wpnsr" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "kube-api-access-wpnsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:47.520732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.520701 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-envoy" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.521133 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.521100 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-token" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:47.521568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.521536 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-podinfo" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.591006 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-certs\") pod \"b1d27dcf-aac3-4773-bc85-20aed144185d\" (UID: \"b1d27dcf-aac3-4773-bc85-20aed144185d\") " Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.591351 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-certs" pod "b1d27dcf-aac3-4773-bc85-20aed144185d" (UID: "b1d27dcf-aac3-4773-bc85-20aed144185d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.591971 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpnsr\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-kube-api-access-wpnsr\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.591997 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.592012 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.592024 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.592037 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.592047 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.592055 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1d27dcf-aac3-4773-bc85-20aed144185d-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.592066 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1d27dcf-aac3-4773-bc85-20aed144185d-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.620698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.592078 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1d27dcf-aac3-4773-bc85-20aed144185d-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:47.891901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.891852 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 04 01:32:47.892089 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.891927 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:32:47.938752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.938725 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" Nov 04 01:32:47.938923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.938750 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr" event={"ID":"b1d27dcf-aac3-4773-bc85-20aed144185d","Type":"ContainerDied","Data":"31fd3f216f27e5c27ee11aa4840e86ce17fd70807d6a029543e3fc9e5a658c12"} Nov 04 01:32:47.938923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.938807 5051 scope.go:122] "RemoveContainer" containerID="661ba69a12888e72adb44d541f75b6598392ac83eb28f1909712a1682f70cba2" Nov 04 01:32:47.996198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.993990 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr"] Nov 04 01:32:47.999726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:47.999694 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-xtrwr"] Nov 04 01:32:48.036088 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:48.036052 5051 scope.go:122] "RemoveContainer" containerID="8e00a8fecfdbfd60d8946471d9e00511c86165468087b52e6c56687b8a0ade47" Nov 04 01:32:48.095240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:48.094988 5051 patch_prober.go:29] interesting pod/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 04 01:32:48.095240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:48.095054 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:32:48.244460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:48.241647 5051 scope.go:122] "RemoveContainer" containerID="51c3d01fee84bae3fe0148dcddc7ed700d3f9988e95fccf46cbe5c3d52080498" Nov 04 01:32:48.333047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:48.333024 5051 scope.go:122] "RemoveContainer" containerID="602338c09187215e2cc3312a2fa8eb4010f0cae79d35b82911b02fe49af653f2" Nov 04 01:32:48.506107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:48.506078 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx"] Nov 04 01:32:48.517638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:48.517604 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p"] Nov 04 01:32:49.360831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:49.360615 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" path="/var/lib/kubelet/pods/b1d27dcf-aac3-4773-bc85-20aed144185d/volumes" Nov 04 01:32:49.583774 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:49.570665 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d6b847_b5dc_45a5_b0b9_1859be473668.slice/crio-8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507aa1e3_d0e1_4c14_ba76_3f271e737658.slice/crio-24288ca1a8c371202d4ee61332ffeba5299b41de612895061b5de9700c5eb36c.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:32:50.201493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.201456 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:50.241869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.238113 5051 generic.go:356] "Generic (PLEG): container finished" podID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerID="8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838" exitCode=0 Nov 04 01:32:50.241869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.238294 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" event={"ID":"61d6b847-b5dc-45a5-b0b9-1859be473668","Type":"ContainerDied","Data":"8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838"} Nov 04 01:32:50.241869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.238349 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" event={"ID":"61d6b847-b5dc-45a5-b0b9-1859be473668","Type":"ContainerDied","Data":"b5e51e48b5fe60cc3eba946cdcb490ea5f95ad0f55bb3dddf06196775f9b147c"} Nov 04 01:32:50.241869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.238401 5051 scope.go:122] "RemoveContainer" containerID="8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838" Nov 04 01:32:50.270391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.245504 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr" Nov 04 01:32:50.350628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.350585 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-kube-api-access-dmdw4\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-kube-api-access-dmdw4\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.350944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.350918 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-socket\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.351232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.351208 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-credential-socket\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.351476 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.351458 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-certs\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.351605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.351590 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-podinfo\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.351904 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.351888 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-envoy\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.355206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.355182 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-data\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.355450 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.355431 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-token\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-token\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.360386 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-certs" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.373655 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-socket" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.373824 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-credential-socket" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.374184 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-data" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.401637 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-kube-api-access-dmdw4" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "kube-api-access-dmdw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.401810 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d6b847-b5dc-45a5-b0b9-1859be473668-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6b847-b5dc-45a5-b0b9-1859be473668-istiod-ca-cert\") pod \"61d6b847-b5dc-45a5-b0b9-1859be473668\" (UID: \"61d6b847-b5dc-45a5-b0b9-1859be473668\") " Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.402792 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdw4\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-kube-api-access-dmdw4\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.402819 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.402834 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.402849 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.408368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.402864 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.454290 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.437807 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-envoy" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.472146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.469567 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d6b847-b5dc-45a5-b0b9-1859be473668-istiod-ca-cert" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:50.485555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.485481 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-token" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:50.486999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.486971 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-podinfo" pod "61d6b847-b5dc-45a5-b0b9-1859be473668" (UID: "61d6b847-b5dc-45a5-b0b9-1859be473668"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:50.600363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.594726 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.600363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.594756 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.600363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.594769 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d6b847-b5dc-45a5-b0b9-1859be473668-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.600363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.594783 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6b847-b5dc-45a5-b0b9-1859be473668-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.718587 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7"] Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.726384 5051 cpu_manager.go:414] "containerMap: removing container" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.726414 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.726441 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a782226f-9d6a-4948-979d-d8efc336678a" containerName="payload-processing" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.726450 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a782226f-9d6a-4948-979d-d8efc336678a" containerName="payload-processing" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.726464 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.726472 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.729251 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5008b044-5916-4244-9e18-a1478be5e552" containerName="payload-pre-processing" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.729272 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5008b044-5916-4244-9e18-a1478be5e552" containerName="payload-pre-processing" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.729791 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" containerName="istio-proxy" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.729817 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a782226f-9d6a-4948-979d-d8efc336678a" containerName="payload-processing" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.729828 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d27dcf-aac3-4773-bc85-20aed144185d" containerName="istio-proxy" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.729842 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5008b044-5916-4244-9e18-a1478be5e552" containerName="payload-pre-processing" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.733980 5051 generic.go:356] "Generic (PLEG): container finished" podID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerID="24288ca1a8c371202d4ee61332ffeba5299b41de612895061b5de9700c5eb36c" exitCode=0 Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.744146 5051 scope.go:122] "RemoveContainer" containerID="8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:50.744974 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838\": container with ID starting with 8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838 not found: ID does not exist" containerID="8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838" Nov 04 01:32:50.745980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.745008 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838"} err="failed to get container status \"8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838\": rpc error: code = NotFound desc = could not find container \"8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838\": container with ID starting with 8b9b30bf346643e8817ceb61389f09e1d094f5671b21657e61a3795240bfa838 not found: ID does not exist" Nov 04 01:32:50.755277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.755249 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7"] Nov 04 01:32:50.755401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.755301 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" event={"ID":"507aa1e3-d0e1-4c14-ba76-3f271e737658","Type":"ContainerDied","Data":"24288ca1a8c371202d4ee61332ffeba5299b41de612895061b5de9700c5eb36c"} Nov 04 01:32:50.755401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.755379 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:50.756732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.756704 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:50.766528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.765077 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:32:50.766528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.765167 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:32:50.766528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.766227 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:32:50.766528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.766279 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:32:50.806995 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.806960 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr"] Nov 04 01:32:50.814344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.814163 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-b6jcr"] Nov 04 01:32:50.920927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.920870 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-kube-api-access-wmlgm\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-kube-api-access-wmlgm\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.920961 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-data\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921168 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-credential-socket\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921243 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507aa1e3-d0e1-4c14-ba76-3f271e737658-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507aa1e3-d0e1-4c14-ba76-3f271e737658-istiod-ca-cert\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921281 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-envoy\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921411 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921362 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-data" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.921411 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921396 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-certs\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921459 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-token\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-token\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921513 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-socket\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921632 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-podinfo\") pod \"507aa1e3-d0e1-4c14-ba76-3f271e737658\" (UID: \"507aa1e3-d0e1-4c14-ba76-3f271e737658\") " Nov 04 01:32:50.921770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921717 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-certs" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.921770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921752 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507aa1e3-d0e1-4c14-ba76-3f271e737658-istiod-ca-cert" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:50.921984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.921951 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:50.922115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922002 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:50.922115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922009 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-socket" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.922448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922424 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-credential-socket" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.922624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922603 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85vs\" (UniqueName: \"kubernetes.io/projected/d92673cc-c7ba-471c-a50d-a9864552e991-kube-api-access-s85vs\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:50.922877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922749 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.922877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922770 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.922877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922785 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507aa1e3-d0e1-4c14-ba76-3f271e737658-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.922877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922799 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.922877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.922811 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:50.924203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.924181 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-token" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:50.924405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.924383 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-envoy" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:50.924506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.924460 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-kube-api-access-wmlgm" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "kube-api-access-wmlgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:50.924704 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:50.924682 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-podinfo" pod "507aa1e3-d0e1-4c14-ba76-3f271e737658" (UID: "507aa1e3-d0e1-4c14-ba76-3f271e737658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.026582 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85vs\" (UniqueName: \"kubernetes.io/projected/d92673cc-c7ba-471c-a50d-a9864552e991-kube-api-access-s85vs\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.026686 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.026724 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.027001 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.027023 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.027060 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmlgm\" (UniqueName: \"kubernetes.io/projected/507aa1e3-d0e1-4c14-ba76-3f271e737658-kube-api-access-wmlgm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.027076 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507aa1e3-d0e1-4c14-ba76-3f271e737658-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.027889 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:51.051645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.035963 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:51.058783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.058744 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85vs\" (UniqueName: \"kubernetes.io/projected/d92673cc-c7ba-471c-a50d-a9864552e991-kube-api-access-s85vs\") pod \"maas-api-revoke-sub-d2c0897e-z87g7\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:51.124833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.124429 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:51.280957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.280894 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d6b847-b5dc-45a5-b0b9-1859be473668" path="/var/lib/kubelet/pods/61d6b847-b5dc-45a5-b0b9-1859be473668/volumes" Nov 04 01:32:51.774146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.758221 5051 patch_prober.go:29] interesting pod/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:32:51.774146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.758302 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:32:51.805988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.803895 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7"] Nov 04 01:32:51.850407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.850361 5051 generic.go:356] "Generic (PLEG): container finished" podID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerID="5aa5f251ad4a72250ea7ce7c0b936a6ac23f0865ea6ab07554508a7e5114fb6e" exitCode=0 Nov 04 01:32:51.855141 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.850707 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" event={"ID":"6897ee6f-4599-48f0-b7d9-3b6484c955bc","Type":"ContainerDied","Data":"5aa5f251ad4a72250ea7ce7c0b936a6ac23f0865ea6ab07554508a7e5114fb6e"} Nov 04 01:32:51.889146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.866187 5051 scope.go:122] "RemoveContainer" containerID="5aa5f251ad4a72250ea7ce7c0b936a6ac23f0865ea6ab07554508a7e5114fb6e" Nov 04 01:32:51.937197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.936227 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:51.937197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.936376 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:51.937197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:51.936436 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:52.069563 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:52.069524 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" event={"ID":"d92673cc-c7ba-471c-a50d-a9864552e991","Type":"ContainerStarted","Data":"ebf382a6d1345f294a19f15c7797ceee532d2acbd44c3d7c9ff81d51aceeb1b6"} Nov 04 01:32:52.101866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:52.075029 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" event={"ID":"507aa1e3-d0e1-4c14-ba76-3f271e737658","Type":"ContainerDied","Data":"eb40088e124d006dc4a402b7fc484de95d12a98c0dae69c5bbeae4c648e25535"} Nov 04 01:32:52.101866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:52.075102 5051 scope.go:122] "RemoveContainer" containerID="24288ca1a8c371202d4ee61332ffeba5299b41de612895061b5de9700c5eb36c" Nov 04 01:32:52.101866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:52.075289 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q" Nov 04 01:32:52.149798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:52.149767 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q"] Nov 04 01:32:52.166722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:52.166692 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3888c434-openshift-default-75b6d654bf-v475q"] Nov 04 01:32:52.993180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:52.992806 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:32:53.098317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.098270 5051 generic.go:356] "Generic (PLEG): container finished" podID="3365fd0a-a069-4da8-bd68-b323fb2411ca" containerID="7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79" exitCode=0 Nov 04 01:32:53.098509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.098484 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" event={"ID":"3365fd0a-a069-4da8-bd68-b323fb2411ca","Type":"ContainerDied","Data":"7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79"} Nov 04 01:32:53.098586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.098543 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" event={"ID":"3365fd0a-a069-4da8-bd68-b323fb2411ca","Type":"ContainerDied","Data":"91f72e716791aa533a8e85064bcb00f46a2cb7156e592bd8d2bbdf2bdbcfced3"} Nov 04 01:32:53.098586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.098577 5051 scope.go:122] "RemoveContainer" containerID="7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79" Nov 04 01:32:53.098722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.098702 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9" Nov 04 01:32:53.121633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.121547 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" event={"ID":"6897ee6f-4599-48f0-b7d9-3b6484c955bc","Type":"ContainerStarted","Data":"31741d3427938c52444f38c23b59ecaa70ba7bf772c346b844883394423ae8d4"} Nov 04 01:32:53.124723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.124382 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" event={"ID":"d92673cc-c7ba-471c-a50d-a9864552e991","Type":"ContainerStarted","Data":"f1e73be98e8e4931d19162b7f21b47551b95e6c5f86a4ea200af933d75c6ea43"} Nov 04 01:32:53.128676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128222 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-credential-socket\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.128676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128292 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-data\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.128676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128341 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-socket\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.128676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128376 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-podinfo\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.128676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128536 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-kube-api-access-4prln\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-kube-api-access-4prln\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.128676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128574 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-envoy\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.129110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128684 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-token\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-token\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.129110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128704 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-socket" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:53.129110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128739 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3365fd0a-a069-4da8-bd68-b323fb2411ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3365fd0a-a069-4da8-bd68-b323fb2411ca-istiod-ca-cert\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.129110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.128783 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-certs\") pod \"3365fd0a-a069-4da8-bd68-b323fb2411ca\" (UID: \"3365fd0a-a069-4da8-bd68-b323fb2411ca\") " Nov 04 01:32:53.129650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.129436 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-data" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:53.130033 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.129756 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.130033 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.129785 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.130033 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.129883 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3365fd0a-a069-4da8-bd68-b323fb2411ca-istiod-ca-cert" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:53.130033 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.130010 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-certs" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:53.131262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.131232 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-credential-socket" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:53.135508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.133006 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-kube-api-access-4prln" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "kube-api-access-4prln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:53.135508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.133286 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-envoy" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:32:53.135508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.133450 5051 scope.go:122] "RemoveContainer" containerID="7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79" Nov 04 01:32:53.136461 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.136435 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-token" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:53.136799 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:53.136777 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79\": container with ID starting with 7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79 not found: ID does not exist" containerID="7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79" Nov 04 01:32:53.136884 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.136810 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79"} err="failed to get container status \"7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79\": rpc error: code = NotFound desc = could not find container \"7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79\": container with ID starting with 7937e9e10ce8e93259b5ef4dc0e46d6ffeab0f10f2a1c3f9fdf9611c6e43be79 not found: ID does not exist" Nov 04 01:32:53.139851 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.139835 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-podinfo" pod "3365fd0a-a069-4da8-bd68-b323fb2411ca" (UID: "3365fd0a-a069-4da8-bd68-b323fb2411ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:32:53.214370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.214257 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" podStartSLOduration=3.214242657 podStartE2EDuration="3.214242657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:32:53.209402113 +0000 UTC m=+2112.181150983" watchObservedRunningTime="2026-11-04 01:32:53.214242657 +0000 UTC m=+2112.185991569" Nov 04 01:32:53.219915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.219883 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" path="/var/lib/kubelet/pods/507aa1e3-d0e1-4c14-ba76-3f271e737658/volumes" Nov 04 01:32:53.231246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.231220 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.231246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.231246 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.231366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.231256 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4prln\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-kube-api-access-4prln\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.231366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.231265 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.231366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.231274 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3365fd0a-a069-4da8-bd68-b323fb2411ca-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.231366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.231283 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3365fd0a-a069-4da8-bd68-b323fb2411ca-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.231366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.231291 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3365fd0a-a069-4da8-bd68-b323fb2411ca-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:53.423207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.422982 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9"] Nov 04 01:32:53.428378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.428346 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-50dab0-openshift-default-56fb4d8b47-5grx9"] Nov 04 01:32:53.507180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.506933 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" podUID="5c8c1b3e-1289-4c31-89d0-e43168ec9619" containerName="payload-processing" containerID="cri-o://741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39" gracePeriod=25 Nov 04 01:32:53.518763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:53.518685 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" podUID="5b86f87c-1d12-4718-b03f-a32375cf87a5" containerName="payload-pre-processing" containerID="cri-o://bcde942b5da34a582b28c4c96c6cca53d4e9cb3a0063bfe1b69fafee331a6fb2" gracePeriod=25 Nov 04 01:32:54.017007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.016489 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:32:54.017007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.016659 5051 patch_prober.go:29] interesting pod/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 04 01:32:54.017007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.016717 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 04 01:32:54.062154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.059212 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:32:54.139916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.139877 5051 generic.go:356] "Generic (PLEG): container finished" podID="5b86f87c-1d12-4718-b03f-a32375cf87a5" containerID="bcde942b5da34a582b28c4c96c6cca53d4e9cb3a0063bfe1b69fafee331a6fb2" exitCode=0 Nov 04 01:32:54.140179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.139992 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" event={"ID":"5b86f87c-1d12-4718-b03f-a32375cf87a5","Type":"ContainerDied","Data":"bcde942b5da34a582b28c4c96c6cca53d4e9cb3a0063bfe1b69fafee331a6fb2"} Nov 04 01:32:54.140179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.140041 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" event={"ID":"5b86f87c-1d12-4718-b03f-a32375cf87a5","Type":"ContainerDied","Data":"e90da0a459651a78c9973c784ffe245ae5707f99a7e419a2495efdb66c3cbbc2"} Nov 04 01:32:54.140179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.140064 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e90da0a459651a78c9973c784ffe245ae5707f99a7e419a2495efdb66c3cbbc2" Nov 04 01:32:54.144583 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.144542 5051 generic.go:356] "Generic (PLEG): container finished" podID="5c8c1b3e-1289-4c31-89d0-e43168ec9619" containerID="741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39" exitCode=0 Nov 04 01:32:54.144760 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.144740 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" Nov 04 01:32:54.144872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.144852 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" event={"ID":"5c8c1b3e-1289-4c31-89d0-e43168ec9619","Type":"ContainerDied","Data":"741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39"} Nov 04 01:32:54.144997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.144981 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx" event={"ID":"5c8c1b3e-1289-4c31-89d0-e43168ec9619","Type":"ContainerDied","Data":"9bb08f9ac0ed10ac2d747ada126131f541c9d0ead70c4d542d5d1b620b29e931"} Nov 04 01:32:54.145107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.145095 5051 scope.go:122] "RemoveContainer" containerID="741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39" Nov 04 01:32:54.148176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.147387 5051 generic.go:356] "Generic (PLEG): container finished" podID="d92673cc-c7ba-471c-a50d-a9864552e991" containerID="f1e73be98e8e4931d19162b7f21b47551b95e6c5f86a4ea200af933d75c6ea43" exitCode=0 Nov 04 01:32:54.148176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.147691 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" event={"ID":"d92673cc-c7ba-471c-a50d-a9864552e991","Type":"ContainerDied","Data":"f1e73be98e8e4931d19162b7f21b47551b95e6c5f86a4ea200af933d75c6ea43"} Nov 04 01:32:54.166144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.163511 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:32:54.275306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.248429 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c8c1b3e-1289-4c31-89d0-e43168ec9619-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8c1b3e-1289-4c31-89d0-e43168ec9619-plugins-config-volume\") pod \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\" (UID: \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\") " Nov 04 01:32:54.275306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.248639 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8c1b3e-1289-4c31-89d0-e43168ec9619-kube-api-access-7q7hd\" (UniqueName: \"kubernetes.io/projected/5c8c1b3e-1289-4c31-89d0-e43168ec9619-kube-api-access-7q7hd\") pod \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\" (UID: \"5c8c1b3e-1289-4c31-89d0-e43168ec9619\") " Nov 04 01:32:54.275306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.268419 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8c1b3e-1289-4c31-89d0-e43168ec9619-plugins-config-volume" pod "5c8c1b3e-1289-4c31-89d0-e43168ec9619" (UID: "5c8c1b3e-1289-4c31-89d0-e43168ec9619"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:54.391351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.374837 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b86f87c-1d12-4718-b03f-a32375cf87a5-kube-api-access-4fjvb\" (UniqueName: \"kubernetes.io/projected/5b86f87c-1d12-4718-b03f-a32375cf87a5-kube-api-access-4fjvb\") pod \"5b86f87c-1d12-4718-b03f-a32375cf87a5\" (UID: \"5b86f87c-1d12-4718-b03f-a32375cf87a5\") " Nov 04 01:32:54.391351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.375108 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b86f87c-1d12-4718-b03f-a32375cf87a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b86f87c-1d12-4718-b03f-a32375cf87a5-plugins-config-volume\") pod \"5b86f87c-1d12-4718-b03f-a32375cf87a5\" (UID: \"5b86f87c-1d12-4718-b03f-a32375cf87a5\") " Nov 04 01:32:54.391351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.375853 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8c1b3e-1289-4c31-89d0-e43168ec9619-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:54.404145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.395335 5051 scope.go:122] "RemoveContainer" containerID="741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39" Nov 04 01:32:54.404940 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:32:54.404868 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39\": container with ID starting with 741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39 not found: ID does not exist" containerID="741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39" Nov 04 01:32:54.404940 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.404904 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39"} err="failed to get container status \"741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39\": rpc error: code = NotFound desc = could not find container \"741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39\": container with ID starting with 741761109fa117946be22a625e5abc08d691ed283adc343f43f7c418f866fc39 not found: ID does not exist" Nov 04 01:32:54.410832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.410792 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b86f87c-1d12-4718-b03f-a32375cf87a5-plugins-config-volume" pod "5b86f87c-1d12-4718-b03f-a32375cf87a5" (UID: "5b86f87c-1d12-4718-b03f-a32375cf87a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:54.411010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.410911 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8c1b3e-1289-4c31-89d0-e43168ec9619-kube-api-access-7q7hd" pod "5c8c1b3e-1289-4c31-89d0-e43168ec9619" (UID: "5c8c1b3e-1289-4c31-89d0-e43168ec9619"). InnerVolumeSpecName "kube-api-access-7q7hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:54.416860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.416823 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b86f87c-1d12-4718-b03f-a32375cf87a5-kube-api-access-4fjvb" pod "5b86f87c-1d12-4718-b03f-a32375cf87a5" (UID: "5b86f87c-1d12-4718-b03f-a32375cf87a5"). InnerVolumeSpecName "kube-api-access-4fjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:54.491995 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.483211 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q7hd\" (UniqueName: \"kubernetes.io/projected/5c8c1b3e-1289-4c31-89d0-e43168ec9619-kube-api-access-7q7hd\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:54.491995 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.483251 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b86f87c-1d12-4718-b03f-a32375cf87a5-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:54.491995 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.483268 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fjvb\" (UniqueName: \"kubernetes.io/projected/5b86f87c-1d12-4718-b03f-a32375cf87a5-kube-api-access-4fjvb\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:54.523040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.523008 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx"] Nov 04 01:32:54.530539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:54.530469 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-75f7ae-684797f94-br2jx"] Nov 04 01:32:55.026396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.026365 5051 patch_prober.go:29] interesting pod/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 04 01:32:55.027286 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.026966 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 04 01:32:55.196618 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.196580 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p" Nov 04 01:32:55.215435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.215403 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3365fd0a-a069-4da8-bd68-b323fb2411ca" path="/var/lib/kubelet/pods/3365fd0a-a069-4da8-bd68-b323fb2411ca/volumes" Nov 04 01:32:55.216282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.216253 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8c1b3e-1289-4c31-89d0-e43168ec9619" path="/var/lib/kubelet/pods/5c8c1b3e-1289-4c31-89d0-e43168ec9619/volumes" Nov 04 01:32:55.231856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.231822 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p"] Nov 04 01:32:55.236703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.236675 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-75f7ae-7b859d7899-kb94p"] Nov 04 01:32:55.534079 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.534047 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:55.769637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.758748 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92673cc-c7ba-471c-a50d-a9864552e991-kube-api-access-s85vs\" (UniqueName: \"kubernetes.io/projected/d92673cc-c7ba-471c-a50d-a9864552e991-kube-api-access-s85vs\") pod \"d92673cc-c7ba-471c-a50d-a9864552e991\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " Nov 04 01:32:55.769637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.758878 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-openshift-ca\") pod \"d92673cc-c7ba-471c-a50d-a9864552e991\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " Nov 04 01:32:55.769637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.759075 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-xks-ca\") pod \"d92673cc-c7ba-471c-a50d-a9864552e991\" (UID: \"d92673cc-c7ba-471c-a50d-a9864552e991\") " Nov 04 01:32:55.807345 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.807307 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-openshift-ca" pod "d92673cc-c7ba-471c-a50d-a9864552e991" (UID: "d92673cc-c7ba-471c-a50d-a9864552e991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:32:55.816298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.816265 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92673cc-c7ba-471c-a50d-a9864552e991-kube-api-access-s85vs" pod "d92673cc-c7ba-471c-a50d-a9864552e991" (UID: "d92673cc-c7ba-471c-a50d-a9864552e991"). InnerVolumeSpecName "kube-api-access-s85vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:32:55.816298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.816263 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-xks-ca" pod "d92673cc-c7ba-471c-a50d-a9864552e991" (UID: "d92673cc-c7ba-471c-a50d-a9864552e991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:32:55.891673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.891608 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:55.891673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.891644 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85vs\" (UniqueName: \"kubernetes.io/projected/d92673cc-c7ba-471c-a50d-a9864552e991-kube-api-access-s85vs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:55.891673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:55.891666 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92673cc-c7ba-471c-a50d-a9864552e991-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:32:56.015888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:56.015848 5051 patch_prober.go:29] interesting pod/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 04 01:32:56.016063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:56.015922 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 04 01:32:56.209071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:56.208995 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" Nov 04 01:32:56.209500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:56.208998 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7" event={"ID":"d92673cc-c7ba-471c-a50d-a9864552e991","Type":"ContainerDied","Data":"ebf382a6d1345f294a19f15c7797ceee532d2acbd44c3d7c9ff81d51aceeb1b6"} Nov 04 01:32:56.209500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:56.209152 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebf382a6d1345f294a19f15c7797ceee532d2acbd44c3d7c9ff81d51aceeb1b6" Nov 04 01:32:57.022752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:57.022298 5051 patch_prober.go:29] interesting pod/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 04 01:32:57.022752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:57.022362 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 04 01:32:57.199926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:57.199888 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b86f87c-1d12-4718-b03f-a32375cf87a5" path="/var/lib/kubelet/pods/5b86f87c-1d12-4718-b03f-a32375cf87a5/volumes" Nov 04 01:32:58.016348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:58.016315 5051 patch_prober.go:29] interesting pod/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Nov 04 01:32:58.016773 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:58.016363 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Nov 04 01:32:58.229319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:58.229274 5051 generic.go:356] "Generic (PLEG): container finished" podID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerID="3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93" exitCode=0 Nov 04 01:32:58.229319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:58.229306 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" event={"ID":"01f13c29-48fe-41ce-ba7f-5bfe6d073eab","Type":"ContainerDied","Data":"3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93"} Nov 04 01:32:58.229792 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:58.229777 5051 scope.go:122] "RemoveContainer" containerID="3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93" Nov 04 01:32:58.231661 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:58.231628 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:58.231793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:58.231696 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:58.231793 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:58.231729 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:32:59.023593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:59.023557 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:32:59.024220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:59.024198 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:32:59.024993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:59.024974 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:32:59.332491 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:32:59.332407 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" event={"ID":"01f13c29-48fe-41ce-ba7f-5bfe6d073eab","Type":"ContainerStarted","Data":"df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4"} Nov 04 01:33:00.223142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:00.202803 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:33:00.223142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:00.202968 5051 patch_prober.go:29] interesting pod/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:33:00.223142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:00.203031 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 04 01:33:01.034393 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:01.034356 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc"] Nov 04 01:33:01.068187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:01.060731 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp"] Nov 04 01:33:01.202094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:01.202057 5051 patch_prober.go:29] interesting pod/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:33:01.202306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:01.202138 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 04 01:33:02.201962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:02.201924 5051 patch_prober.go:29] interesting pod/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:33:02.202408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:02.201978 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 04 01:33:03.201063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:03.201011 5051 patch_prober.go:29] interesting pod/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:33:03.201305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:03.201082 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 04 01:33:04.108014 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:04.107971 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:33:04.201870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:04.201379 5051 patch_prober.go:29] interesting pod/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:33:04.201870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:04.201455 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 04 01:33:05.201733 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:05.201699 5051 patch_prober.go:29] interesting pod/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 04 01:33:05.202157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:05.201749 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 04 01:33:05.616154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:05.616104 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:33:06.038143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.034819 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" podUID="1530f1ae-087d-4766-aaae-02a5e6531b04" containerName="payload-processing" containerID="cri-o://aa387b00ec92e80d5fcfcc8aa69230456b4db723da5bba06188a8d7a7f5fb704" gracePeriod=25 Nov 04 01:33:06.063784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.063728 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" podUID="3bcb9033-bd09-4223-8655-7d12939824b2" containerName="payload-pre-processing" containerID="cri-o://840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58" gracePeriod=25 Nov 04 01:33:06.224870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.224835 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:33:06.483884 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.483479 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:33:06.500290 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.500264 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bcb9033-bd09-4223-8655-7d12939824b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcb9033-bd09-4223-8655-7d12939824b2-plugins-config-volume\") pod \"3bcb9033-bd09-4223-8655-7d12939824b2\" (UID: \"3bcb9033-bd09-4223-8655-7d12939824b2\") " Nov 04 01:33:06.500451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.500313 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bcb9033-bd09-4223-8655-7d12939824b2-kube-api-access-sksk4\" (UniqueName: \"kubernetes.io/projected/3bcb9033-bd09-4223-8655-7d12939824b2-kube-api-access-sksk4\") pod \"3bcb9033-bd09-4223-8655-7d12939824b2\" (UID: \"3bcb9033-bd09-4223-8655-7d12939824b2\") " Nov 04 01:33:06.500699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.500674 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bcb9033-bd09-4223-8655-7d12939824b2-plugins-config-volume" pod "3bcb9033-bd09-4223-8655-7d12939824b2" (UID: "3bcb9033-bd09-4223-8655-7d12939824b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:06.501107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.501087 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcb9033-bd09-4223-8655-7d12939824b2-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:06.503866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.503841 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcb9033-bd09-4223-8655-7d12939824b2-kube-api-access-sksk4" pod "3bcb9033-bd09-4223-8655-7d12939824b2" (UID: "3bcb9033-bd09-4223-8655-7d12939824b2"). InnerVolumeSpecName "kube-api-access-sksk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:06.534039 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.534011 5051 generic.go:356] "Generic (PLEG): container finished" podID="1530f1ae-087d-4766-aaae-02a5e6531b04" containerID="aa387b00ec92e80d5fcfcc8aa69230456b4db723da5bba06188a8d7a7f5fb704" exitCode=0 Nov 04 01:33:06.534234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.534090 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" event={"ID":"1530f1ae-087d-4766-aaae-02a5e6531b04","Type":"ContainerDied","Data":"aa387b00ec92e80d5fcfcc8aa69230456b4db723da5bba06188a8d7a7f5fb704"} Nov 04 01:33:06.536531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.536499 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:33:06.537329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.537305 5051 generic.go:356] "Generic (PLEG): container finished" podID="3bcb9033-bd09-4223-8655-7d12939824b2" containerID="840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58" exitCode=0 Nov 04 01:33:06.537439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.537340 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" Nov 04 01:33:06.537439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.537366 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" event={"ID":"3bcb9033-bd09-4223-8655-7d12939824b2","Type":"ContainerDied","Data":"840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58"} Nov 04 01:33:06.537439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.537405 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp" event={"ID":"3bcb9033-bd09-4223-8655-7d12939824b2","Type":"ContainerDied","Data":"33975418776fb9300f4609acc554713b24516ff9532495f6d5bc811db76d6a1d"} Nov 04 01:33:06.537439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.537433 5051 scope.go:122] "RemoveContainer" containerID="840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58" Nov 04 01:33:06.540856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.540835 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:33:06.559798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.559706 5051 scope.go:122] "RemoveContainer" containerID="840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58" Nov 04 01:33:06.560673 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:06.560603 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58\": container with ID starting with 840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58 not found: ID does not exist" containerID="840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58" Nov 04 01:33:06.560673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.560637 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58"} err="failed to get container status \"840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58\": rpc error: code = NotFound desc = could not find container \"840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58\": container with ID starting with 840905586f36ccc90f2f156ca5568bd71ca3081e33a62190189e328cd57eee58 not found: ID does not exist" Nov 04 01:33:06.605980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.605832 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sksk4\" (UniqueName: \"kubernetes.io/projected/3bcb9033-bd09-4223-8655-7d12939824b2-kube-api-access-sksk4\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:06.612416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.610336 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp"] Nov 04 01:33:06.630466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.630431 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-385061-7f47588bc5-xrwtp"] Nov 04 01:33:06.707033 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.707002 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1530f1ae-087d-4766-aaae-02a5e6531b04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f1ae-087d-4766-aaae-02a5e6531b04-plugins-config-volume\") pod \"1530f1ae-087d-4766-aaae-02a5e6531b04\" (UID: \"1530f1ae-087d-4766-aaae-02a5e6531b04\") " Nov 04 01:33:06.707308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.707287 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1530f1ae-087d-4766-aaae-02a5e6531b04-kube-api-access-xggd5\" (UniqueName: \"kubernetes.io/projected/1530f1ae-087d-4766-aaae-02a5e6531b04-kube-api-access-xggd5\") pod \"1530f1ae-087d-4766-aaae-02a5e6531b04\" (UID: \"1530f1ae-087d-4766-aaae-02a5e6531b04\") " Nov 04 01:33:06.707476 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.707450 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1530f1ae-087d-4766-aaae-02a5e6531b04-plugins-config-volume" pod "1530f1ae-087d-4766-aaae-02a5e6531b04" (UID: "1530f1ae-087d-4766-aaae-02a5e6531b04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:06.708258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.708237 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1530f1ae-087d-4766-aaae-02a5e6531b04-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:06.709595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.709569 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1530f1ae-087d-4766-aaae-02a5e6531b04-kube-api-access-xggd5" pod "1530f1ae-087d-4766-aaae-02a5e6531b04" (UID: "1530f1ae-087d-4766-aaae-02a5e6531b04"). InnerVolumeSpecName "kube-api-access-xggd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:06.833145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:06.824314 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xggd5\" (UniqueName: \"kubernetes.io/projected/1530f1ae-087d-4766-aaae-02a5e6531b04-kube-api-access-xggd5\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:07.190928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:07.190886 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bcb9033-bd09-4223-8655-7d12939824b2" path="/var/lib/kubelet/pods/3bcb9033-bd09-4223-8655-7d12939824b2/volumes" Nov 04 01:33:07.548863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:07.548833 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" Nov 04 01:33:07.549390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:07.548861 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc" event={"ID":"1530f1ae-087d-4766-aaae-02a5e6531b04","Type":"ContainerDied","Data":"edf6bb32adfeb241320755b8de00c2abc6192456cd7cb07db8e6a5c546977107"} Nov 04 01:33:07.549390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:07.548919 5051 scope.go:122] "RemoveContainer" containerID="aa387b00ec92e80d5fcfcc8aa69230456b4db723da5bba06188a8d7a7f5fb704" Nov 04 01:33:07.587532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:07.587501 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc"] Nov 04 01:33:07.593115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:07.593085 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-385061-dd9df9788-sqqrc"] Nov 04 01:33:09.190877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:09.190835 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1530f1ae-087d-4766-aaae-02a5e6531b04" path="/var/lib/kubelet/pods/1530f1ae-087d-4766-aaae-02a5e6531b04/volumes" Nov 04 01:33:11.670541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:11.670506 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9"] Nov 04 01:33:11.683608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:11.683574 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7"] Nov 04 01:33:16.123186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.123145 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh"] Nov 04 01:33:16.124196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.124157 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" containerID="cri-o://fe29a4a9d21efc11ad44e5b25ae142024a216371d5ddc570981711232098ddf0" gracePeriod=30 Nov 04 01:33:16.127077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.127047 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm"] Nov 04 01:33:16.127287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.127265 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" containerID="cri-o://cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e" gracePeriod=30 Nov 04 01:33:16.134009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.133982 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz"] Nov 04 01:33:16.134247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.134193 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" containerID="cri-o://dbfa6f8f004702dd7b0766d8c27b2b30e1955e146643ec7257855742e2b99d79" gracePeriod=30 Nov 04 01:33:16.536364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.523807 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:16.536744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.536699 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:16.589541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.588225 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6"] Nov 04 01:33:16.589541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.588466 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" containerID="cri-o://c29baaba157ffc827f199126f19f3910f423cb182a6af935e8ef4c437cbc5e73" gracePeriod=30 Nov 04 01:33:16.667153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.659705 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc"] Nov 04 01:33:16.667153 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.659984 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" podUID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" containerName="istio-proxy" containerID="cri-o://f0cf73aa2fc6bd8275026fa45aa75a69a9142753a0e99a91684369a6d820b50c" gracePeriod=30 Nov 04 01:33:16.682147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.680919 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" podUID="17f25302-ecbc-4091-86d9-19234028c089" containerName="payload-processing" containerID="cri-o://9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265" gracePeriod=25 Nov 04 01:33:16.731855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:16.711847 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" podUID="af4e2767-65a2-4f53-b23a-673b34dfd0ad" containerName="payload-pre-processing" containerID="cri-o://e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada" gracePeriod=25 Nov 04 01:33:17.537110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.537080 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:33:17.607419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.607390 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:33:17.619737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.619696 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4e2767-65a2-4f53-b23a-673b34dfd0ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4e2767-65a2-4f53-b23a-673b34dfd0ad-plugins-config-volume\") pod \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\" (UID: \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\") " Nov 04 01:33:17.620334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.620308 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4e2767-65a2-4f53-b23a-673b34dfd0ad-plugins-config-volume" pod "af4e2767-65a2-4f53-b23a-673b34dfd0ad" (UID: "af4e2767-65a2-4f53-b23a-673b34dfd0ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:17.621277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.621254 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af4e2767-65a2-4f53-b23a-673b34dfd0ad-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.722253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.722220 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f25302-ecbc-4091-86d9-19234028c089-kube-api-access-m8mrb\" (UniqueName: \"kubernetes.io/projected/17f25302-ecbc-4091-86d9-19234028c089-kube-api-access-m8mrb\") pod \"17f25302-ecbc-4091-86d9-19234028c089\" (UID: \"17f25302-ecbc-4091-86d9-19234028c089\") " Nov 04 01:33:17.722422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.722279 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4e2767-65a2-4f53-b23a-673b34dfd0ad-kube-api-access-h8z58\" (UniqueName: \"kubernetes.io/projected/af4e2767-65a2-4f53-b23a-673b34dfd0ad-kube-api-access-h8z58\") pod \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\" (UID: \"af4e2767-65a2-4f53-b23a-673b34dfd0ad\") " Nov 04 01:33:17.722422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.722378 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f25302-ecbc-4091-86d9-19234028c089-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f25302-ecbc-4091-86d9-19234028c089-plugins-config-volume\") pod \"17f25302-ecbc-4091-86d9-19234028c089\" (UID: \"17f25302-ecbc-4091-86d9-19234028c089\") " Nov 04 01:33:17.722860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.722835 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f25302-ecbc-4091-86d9-19234028c089-plugins-config-volume" pod "17f25302-ecbc-4091-86d9-19234028c089" (UID: "17f25302-ecbc-4091-86d9-19234028c089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:17.722997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.722975 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f25302-ecbc-4091-86d9-19234028c089-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.724510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.724484 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f25302-ecbc-4091-86d9-19234028c089-kube-api-access-m8mrb" pod "17f25302-ecbc-4091-86d9-19234028c089" (UID: "17f25302-ecbc-4091-86d9-19234028c089"). InnerVolumeSpecName "kube-api-access-m8mrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:17.724634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.724528 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4e2767-65a2-4f53-b23a-673b34dfd0ad-kube-api-access-h8z58" pod "af4e2767-65a2-4f53-b23a-673b34dfd0ad" (UID: "af4e2767-65a2-4f53-b23a-673b34dfd0ad"). InnerVolumeSpecName "kube-api-access-h8z58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:17.780564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.780480 5051 patch_prober.go:29] interesting pod/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:17.780738 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.780630 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:17.824404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.824365 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mrb\" (UniqueName: \"kubernetes.io/projected/17f25302-ecbc-4091-86d9-19234028c089-kube-api-access-m8mrb\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.824404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.824397 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8z58\" (UniqueName: \"kubernetes.io/projected/af4e2767-65a2-4f53-b23a-673b34dfd0ad-kube-api-access-h8z58\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:17.916339 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.916305 5051 generic.go:356] "Generic (PLEG): container finished" podID="af4e2767-65a2-4f53-b23a-673b34dfd0ad" containerID="e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada" exitCode=0 Nov 04 01:33:17.916542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.916372 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" Nov 04 01:33:17.916542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.916392 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" event={"ID":"af4e2767-65a2-4f53-b23a-673b34dfd0ad","Type":"ContainerDied","Data":"e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada"} Nov 04 01:33:17.916542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.916494 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7" event={"ID":"af4e2767-65a2-4f53-b23a-673b34dfd0ad","Type":"ContainerDied","Data":"233310a23bd9c3f054aa5128e0d4b2aa570f39b9f99287e132792926f57f858d"} Nov 04 01:33:17.916542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.916526 5051 scope.go:122] "RemoveContainer" containerID="e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada" Nov 04 01:33:17.917953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.917930 5051 generic.go:356] "Generic (PLEG): container finished" podID="17f25302-ecbc-4091-86d9-19234028c089" containerID="9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265" exitCode=0 Nov 04 01:33:17.918081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.918020 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" Nov 04 01:33:17.918081 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.918024 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" event={"ID":"17f25302-ecbc-4091-86d9-19234028c089","Type":"ContainerDied","Data":"9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265"} Nov 04 01:33:17.918190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.918080 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9" event={"ID":"17f25302-ecbc-4091-86d9-19234028c089","Type":"ContainerDied","Data":"e5fab44523f5ef9e955539e73d4ac14010cb34765dd4e5c6d2504ef4218ab9d4"} Nov 04 01:33:17.930773 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.930757 5051 scope.go:122] "RemoveContainer" containerID="e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada" Nov 04 01:33:17.931576 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:17.931008 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada\": container with ID starting with e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada not found: ID does not exist" containerID="e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada" Nov 04 01:33:17.931576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.931031 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada"} err="failed to get container status \"e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada\": rpc error: code = NotFound desc = could not find container \"e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada\": container with ID starting with e197b1196c68df2c670e10a6135ad5db8a0be9131c8c0849c4abd60a537a2ada not found: ID does not exist" Nov 04 01:33:17.931576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.931048 5051 scope.go:122] "RemoveContainer" containerID="9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265" Nov 04 01:33:17.948261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.948243 5051 scope.go:122] "RemoveContainer" containerID="9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265" Nov 04 01:33:17.948549 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:17.948526 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265\": container with ID starting with 9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265 not found: ID does not exist" containerID="9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265" Nov 04 01:33:17.948611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.948556 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265"} err="failed to get container status \"9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265\": rpc error: code = NotFound desc = could not find container \"9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265\": container with ID starting with 9922e8498ce989aeb9a15be141eed545b43c958924c439b01c1a698fa4849265 not found: ID does not exist" Nov 04 01:33:17.972436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.972177 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9"] Nov 04 01:33:17.992960 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:17.992923 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4f8dd0-7ff64878bb-f59f9"] Nov 04 01:33:18.006223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:18.006186 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7"] Nov 04 01:33:18.010390 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:18.010362 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4f8dd0-5fd987cfcf-w8bk7"] Nov 04 01:33:19.194310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:19.194219 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f25302-ecbc-4091-86d9-19234028c089" path="/var/lib/kubelet/pods/17f25302-ecbc-4091-86d9-19234028c089/volumes" Nov 04 01:33:19.194773 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:19.194729 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4e2767-65a2-4f53-b23a-673b34dfd0ad" path="/var/lib/kubelet/pods/af4e2767-65a2-4f53-b23a-673b34dfd0ad/volumes" Nov 04 01:33:19.444948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:19.444857 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:33:19.444948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:19.444930 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" podUID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:33:20.649155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:20.637172 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44"] Nov 04 01:33:20.763809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:20.763769 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr"] Nov 04 01:33:21.470329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:21.470229 5051 patch_prober.go:29] interesting pod/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:33:21.470329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:21.470307 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:21.759217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:21.759157 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p"] Nov 04 01:33:21.761229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:21.760773 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" containerID="cri-o://31741d3427938c52444f38c23b59ecaa70ba7bf772c346b844883394423ae8d4" gracePeriod=30 Nov 04 01:33:22.131536 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.131506 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:33:22.239587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.239528 5051 generic.go:356] "Generic (PLEG): container finished" podID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerID="c29baaba157ffc827f199126f19f3910f423cb182a6af935e8ef4c437cbc5e73" exitCode=0 Nov 04 01:33:22.239947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.239917 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" event={"ID":"747c4b7d-4c96-42ab-b459-8eb5aaf75e09","Type":"ContainerDied","Data":"c29baaba157ffc827f199126f19f3910f423cb182a6af935e8ef4c437cbc5e73"} Nov 04 01:33:22.240166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.240149 5051 scope.go:122] "RemoveContainer" containerID="5170c75c442c95a9d06c312d180443dcf6dec2ec9e051c9b36e05411e4c910fc" Nov 04 01:33:22.243845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.243528 5051 generic.go:356] "Generic (PLEG): container finished" podID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" containerID="f0cf73aa2fc6bd8275026fa45aa75a69a9142753a0e99a91684369a6d820b50c" exitCode=0 Nov 04 01:33:22.243845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.243729 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" event={"ID":"9540afba-e788-48cf-9e8e-b8a685b2f8b6","Type":"ContainerDied","Data":"f0cf73aa2fc6bd8275026fa45aa75a69a9142753a0e99a91684369a6d820b50c"} Nov 04 01:33:22.246469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.245892 5051 generic.go:356] "Generic (PLEG): container finished" podID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerID="cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e" exitCode=0 Nov 04 01:33:22.246469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.245968 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" event={"ID":"70e34ad9-7fee-457e-9a62-98badb115fb0","Type":"ContainerDied","Data":"cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e"} Nov 04 01:33:22.246469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.246009 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" event={"ID":"70e34ad9-7fee-457e-9a62-98badb115fb0","Type":"ContainerDied","Data":"db2b8d31937d268f9957db1cd1aa4690a3902c98b4d3a4da92bb9cae54ccbb58"} Nov 04 01:33:22.246469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.246096 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm" Nov 04 01:33:22.256032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.255904 5051 generic.go:356] "Generic (PLEG): container finished" podID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerID="dbfa6f8f004702dd7b0766d8c27b2b30e1955e146643ec7257855742e2b99d79" exitCode=0 Nov 04 01:33:22.267428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.262187 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" event={"ID":"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020","Type":"ContainerDied","Data":"dbfa6f8f004702dd7b0766d8c27b2b30e1955e146643ec7257855742e2b99d79"} Nov 04 01:33:22.267428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.266436 5051 generic.go:356] "Generic (PLEG): container finished" podID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerID="fe29a4a9d21efc11ad44e5b25ae142024a216371d5ddc570981711232098ddf0" exitCode=0 Nov 04 01:33:22.267428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.266549 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" event={"ID":"3631a39e-5bb5-4de8-9fa2-2830f3f00a94","Type":"ContainerDied","Data":"fe29a4a9d21efc11ad44e5b25ae142024a216371d5ddc570981711232098ddf0"} Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.272050 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-credential-socket\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.272221 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-token\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-token\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.272309 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-data\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.272431 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-envoy\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.272532 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-kube-api-access-wd6gb\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-kube-api-access-wd6gb\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.272592 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e34ad9-7fee-457e-9a62-98badb115fb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e34ad9-7fee-457e-9a62-98badb115fb0-istiod-ca-cert\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.272779 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-certs\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.272918 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-podinfo\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.273072 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-socket\") pod \"70e34ad9-7fee-457e-9a62-98badb115fb0\" (UID: \"70e34ad9-7fee-457e-9a62-98badb115fb0\") " Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.274474 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-socket" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.274901 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-credential-socket" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.278056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.277972 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-token" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.278781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.278259 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-data" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.279409 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.279092 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-certs" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.280610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.280583 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e34ad9-7fee-457e-9a62-98badb115fb0-istiod-ca-cert" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:22.319360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.295672 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-podinfo" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:22.319360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.297275 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-envoy" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.319360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.306240 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-kube-api-access-wd6gb" pod "70e34ad9-7fee-457e-9a62-98badb115fb0" (UID: "70e34ad9-7fee-457e-9a62-98badb115fb0"). InnerVolumeSpecName "kube-api-access-wd6gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.335412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.335391 5051 scope.go:122] "RemoveContainer" containerID="cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e" Nov 04 01:33:22.369817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.369377 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375791 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375824 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375838 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375854 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375867 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375882 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd6gb\" (UniqueName: \"kubernetes.io/projected/70e34ad9-7fee-457e-9a62-98badb115fb0-kube-api-access-wd6gb\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375896 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70e34ad9-7fee-457e-9a62-98badb115fb0-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375910 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70e34ad9-7fee-457e-9a62-98badb115fb0-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.376202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.375923 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70e34ad9-7fee-457e-9a62-98badb115fb0-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.396396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.396314 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:33:22.405063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.404759 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:33:22.407803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.406230 5051 scope.go:122] "RemoveContainer" containerID="cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e" Nov 04 01:33:22.407803 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:22.406529 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e\": container with ID starting with cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e not found: ID does not exist" containerID="cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e" Nov 04 01:33:22.407803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.406563 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e"} err="failed to get container status \"cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e\": rpc error: code = NotFound desc = could not find container \"cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e\": container with ID starting with cdde691c5bbbe14a19bfddfb23df6f9189fdcc876bd200cb0a6874e46fdfe45e not found: ID does not exist" Nov 04 01:33:22.407803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.406586 5051 scope.go:122] "RemoveContainer" containerID="2f0ffce8ce0c12d65173062cd7d74c4a1fccde75ebb2da2355d7669843e7ab40" Nov 04 01:33:22.429242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.429216 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:33:22.476970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.476938 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istiod-ca-cert\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.476977 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-token\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-token\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477001 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-data\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477064 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-credential-socket\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477107 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-socket\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477197 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-kube-api-access-drcgq\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-kube-api-access-drcgq\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477392 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-data" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.477422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477402 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istiod-ca-cert" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:22.477605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477410 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-envoy\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477489 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-podinfo\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477649 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-socket" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.477705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477676 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-certs\") pod \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\" (UID: \"747c4b7d-4c96-42ab-b459-8eb5aaf75e09\") " Nov 04 01:33:22.477835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.477818 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-credential-socket" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.478549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.478445 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-certs" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.478549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.478450 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.478549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.478497 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.478549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.478515 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.478549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.478529 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.481444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.481414 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-podinfo" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:22.481986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.481965 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-envoy" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.482265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.482247 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-kube-api-access-drcgq" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "kube-api-access-drcgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.482448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.482424 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-token" pod "747c4b7d-4c96-42ab-b459-8eb5aaf75e09" (UID: "747c4b7d-4c96-42ab-b459-8eb5aaf75e09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.544050 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.543957 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88"] Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.580872 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-podinfo\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.580942 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istiod-ca-cert\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581001 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-envoy\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581034 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-token\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-token\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581077 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-podinfo\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581113 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-data\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581164 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-certs\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581191 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-kube-api-access-gqvnh\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-kube-api-access-gqvnh\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581242 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-envoy\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581354 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-token\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-token\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581383 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-credential-socket\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581499 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-kube-api-access-qvvdm\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-kube-api-access-qvvdm\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581567 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-credential-socket\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581618 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istiod-ca-cert\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581661 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-certs\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581697 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-certs\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581759 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-data\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581803 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-envoy\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581883 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-podinfo\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581926 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istiod-ca-cert\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.581978 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-kube-api-access-n54pc\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-kube-api-access-n54pc\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.582008 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-credential-socket\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.582076 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-data\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.582131 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-socket\") pod \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\" (UID: \"9540afba-e788-48cf-9e8e-b8a685b2f8b6\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.582239 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-socket\") pod \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\" (UID: \"3631a39e-5bb5-4de8-9fa2-2830f3f00a94\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.582268 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-token\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-token\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.582313 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-socket\") pod \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\" (UID: \"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020\") " Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.583268 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.583296 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.583315 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drcgq\" (UniqueName: \"kubernetes.io/projected/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-kube-api-access-drcgq\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.583332 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.583348 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747c4b7d-4c96-42ab-b459-8eb5aaf75e09-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.587027 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-certs" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.587346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.587038 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-data" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.588345 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-credential-socket" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.588535 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-data" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.588864 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-socket" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.589157 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istiod-ca-cert" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.589301 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istiod-ca-cert" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.589347 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-certs" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.589925 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-data" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.589973 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-socket" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.590029 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-envoy" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.590322 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-kube-api-access-n54pc" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "kube-api-access-n54pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.590351 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-certs" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.590404 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-credential-socket" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.591746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.590636 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-credential-socket" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.593610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.592815 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istiod-ca-cert" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:22.593751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.593730 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf"] Nov 04 01:33:22.594376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.594354 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-socket" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.597505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.597352 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-podinfo" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:22.603235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.601438 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-kube-api-access-gqvnh" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "kube-api-access-gqvnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.616159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.604431 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-envoy" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.616159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.604737 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-token" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.624145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.624064 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-kube-api-access-qvvdm" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "kube-api-access-qvvdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.624145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.624068 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-podinfo" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:22.624332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.624254 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-envoy" pod "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" (UID: "46a8fc37-9b81-44e3-ae6e-6d1c3b86b020"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:22.624471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.624412 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-podinfo" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:22.626637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.626588 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-token" pod "9540afba-e788-48cf-9e8e-b8a685b2f8b6" (UID: "9540afba-e788-48cf-9e8e-b8a685b2f8b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.628547 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.628522 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-token" pod "3631a39e-5bb5-4de8-9fa2-2830f3f00a94" (UID: "3631a39e-5bb5-4de8-9fa2-2830f3f00a94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:22.640620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.640564 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm"] Nov 04 01:33:22.647795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.647767 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-kc2hm"] Nov 04 01:33:22.685012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.684977 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685013 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685031 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685045 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685058 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685076 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685091 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqvnh\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-kube-api-access-gqvnh\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685103 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685112 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685137 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685149 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvvdm\" (UniqueName: \"kubernetes.io/projected/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-kube-api-access-qvvdm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685160 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685171 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685180 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685189 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685198 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685206 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685215 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685224 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685234 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n54pc\" (UniqueName: \"kubernetes.io/projected/9540afba-e788-48cf-9e8e-b8a685b2f8b6-kube-api-access-n54pc\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685244 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685252 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685260 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9540afba-e788-48cf-9e8e-b8a685b2f8b6-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685268 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3631a39e-5bb5-4de8-9fa2-2830f3f00a94-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.685270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685276 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.686060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685285 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:22.686060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:22.685295 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9540afba-e788-48cf-9e8e-b8a685b2f8b6-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:23.200016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.199983 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" path="/var/lib/kubelet/pods/70e34ad9-7fee-457e-9a62-98badb115fb0/volumes" Nov 04 01:33:23.277714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.276880 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" Nov 04 01:33:23.277714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.276889 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6" event={"ID":"747c4b7d-4c96-42ab-b459-8eb5aaf75e09","Type":"ContainerDied","Data":"765194c514c2e5d8a73d9174772fffa7062c1317e32cedc504fd1995ac86e257"} Nov 04 01:33:23.277714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.277011 5051 scope.go:122] "RemoveContainer" containerID="c29baaba157ffc827f199126f19f3910f423cb182a6af935e8ef4c437cbc5e73" Nov 04 01:33:23.280069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.279097 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" Nov 04 01:33:23.280069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.279111 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc" event={"ID":"9540afba-e788-48cf-9e8e-b8a685b2f8b6","Type":"ContainerDied","Data":"f7ec83c532aeff3d6fc1039b9967fef85c92cb842391e8ff6fb1fce588cca0f0"} Nov 04 01:33:23.283235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.283210 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" event={"ID":"46a8fc37-9b81-44e3-ae6e-6d1c3b86b020","Type":"ContainerDied","Data":"35f5725d91a7d15215bea0ba524bd85f825e90747d5e0ff35fa970c165981523"} Nov 04 01:33:23.283332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.283298 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz" Nov 04 01:33:23.286718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.285981 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" event={"ID":"3631a39e-5bb5-4de8-9fa2-2830f3f00a94","Type":"ContainerDied","Data":"06bd316ae8b402b1520702f5151e9103914f67594c2732a6e3584fc6f69ac426"} Nov 04 01:33:23.286718 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.286192 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh" Nov 04 01:33:23.296149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.296101 5051 scope.go:122] "RemoveContainer" containerID="f0cf73aa2fc6bd8275026fa45aa75a69a9142753a0e99a91684369a6d820b50c" Nov 04 01:33:23.308894 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.308872 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6"] Nov 04 01:33:23.312612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.312595 5051 scope.go:122] "RemoveContainer" containerID="dbfa6f8f004702dd7b0766d8c27b2b30e1955e146643ec7257855742e2b99d79" Nov 04 01:33:23.323863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.323840 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-8fjd6"] Nov 04 01:33:23.327289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.327268 5051 scope.go:122] "RemoveContainer" containerID="fe29a4a9d21efc11ad44e5b25ae142024a216371d5ddc570981711232098ddf0" Nov 04 01:33:23.359442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.359412 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc"] Nov 04 01:33:23.365006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.364978 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-dplwc"] Nov 04 01:33:23.382759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.382729 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz"] Nov 04 01:33:23.390160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.390110 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-rbxxz"] Nov 04 01:33:23.417183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.417149 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh"] Nov 04 01:33:23.426003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:23.425977 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-75f7ae-openshift-default-57d8897c77-q7mhh"] Nov 04 01:33:25.194073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.193979 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" path="/var/lib/kubelet/pods/3631a39e-5bb5-4de8-9fa2-2830f3f00a94/volumes" Nov 04 01:33:25.194810 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.194787 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" path="/var/lib/kubelet/pods/46a8fc37-9b81-44e3-ae6e-6d1c3b86b020/volumes" Nov 04 01:33:25.195388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.195372 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" path="/var/lib/kubelet/pods/747c4b7d-4c96-42ab-b459-8eb5aaf75e09/volumes" Nov 04 01:33:25.196616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.196594 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" path="/var/lib/kubelet/pods/9540afba-e788-48cf-9e8e-b8a685b2f8b6/volumes" Nov 04 01:33:25.395194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.394519 5051 generic.go:356] "Generic (PLEG): container finished" podID="5146a1c5-733e-4454-a89d-1071139316c1" containerID="3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3" exitCode=0 Nov 04 01:33:25.395194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.394717 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" event={"ID":"5146a1c5-733e-4454-a89d-1071139316c1","Type":"ContainerDied","Data":"3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3"} Nov 04 01:33:25.395543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.395282 5051 scope.go:122] "RemoveContainer" containerID="3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3" Nov 04 01:33:25.405292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.397585 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:25.405292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.397683 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:25.405292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.397728 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:25.638060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.638021 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" podUID="77f531bf-f087-4ccc-a582-91a0a87e04a7" containerName="payload-processing" containerID="cri-o://ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea" gracePeriod=25 Nov 04 01:33:25.764687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:25.764186 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" podUID="46690fdf-fd41-4366-b213-a60d37e78b45" containerName="payload-pre-processing" containerID="cri-o://4fda571039a8f949c7bf414f33a8cbfadd28ca0f62620604d3750e018fc78a12" gracePeriod=25 Nov 04 01:33:26.017977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.017571 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:33:26.173270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.172434 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f531bf-f087-4ccc-a582-91a0a87e04a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f531bf-f087-4ccc-a582-91a0a87e04a7-plugins-config-volume\") pod \"77f531bf-f087-4ccc-a582-91a0a87e04a7\" (UID: \"77f531bf-f087-4ccc-a582-91a0a87e04a7\") " Nov 04 01:33:26.173270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.172644 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f531bf-f087-4ccc-a582-91a0a87e04a7-kube-api-access-ll6zx\" (UniqueName: \"kubernetes.io/projected/77f531bf-f087-4ccc-a582-91a0a87e04a7-kube-api-access-ll6zx\") pod \"77f531bf-f087-4ccc-a582-91a0a87e04a7\" (UID: \"77f531bf-f087-4ccc-a582-91a0a87e04a7\") " Nov 04 01:33:26.173270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.173148 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f531bf-f087-4ccc-a582-91a0a87e04a7-plugins-config-volume" pod "77f531bf-f087-4ccc-a582-91a0a87e04a7" (UID: "77f531bf-f087-4ccc-a582-91a0a87e04a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:26.177174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.173575 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f531bf-f087-4ccc-a582-91a0a87e04a7-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:26.194603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.183355 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f531bf-f087-4ccc-a582-91a0a87e04a7-kube-api-access-ll6zx" pod "77f531bf-f087-4ccc-a582-91a0a87e04a7" (UID: "77f531bf-f087-4ccc-a582-91a0a87e04a7"). InnerVolumeSpecName "kube-api-access-ll6zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:26.296906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.296769 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll6zx\" (UniqueName: \"kubernetes.io/projected/77f531bf-f087-4ccc-a582-91a0a87e04a7-kube-api-access-ll6zx\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:26.421909 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.421877 5051 generic.go:356] "Generic (PLEG): container finished" podID="77f531bf-f087-4ccc-a582-91a0a87e04a7" containerID="ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea" exitCode=0 Nov 04 01:33:26.422075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.421951 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" event={"ID":"77f531bf-f087-4ccc-a582-91a0a87e04a7","Type":"ContainerDied","Data":"ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea"} Nov 04 01:33:26.422075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.421991 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" event={"ID":"77f531bf-f087-4ccc-a582-91a0a87e04a7","Type":"ContainerDied","Data":"407f0b21ae34a5de083aa221cc4f21614be67642b628265f4ddec63e31170cde"} Nov 04 01:33:26.422075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.422011 5051 scope.go:122] "RemoveContainer" containerID="ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea" Nov 04 01:33:26.422353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.422104 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44" Nov 04 01:33:26.427323 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.427295 5051 generic.go:356] "Generic (PLEG): container finished" podID="46690fdf-fd41-4366-b213-a60d37e78b45" containerID="4fda571039a8f949c7bf414f33a8cbfadd28ca0f62620604d3750e018fc78a12" exitCode=0 Nov 04 01:33:26.427577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.427556 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" event={"ID":"46690fdf-fd41-4366-b213-a60d37e78b45","Type":"ContainerDied","Data":"4fda571039a8f949c7bf414f33a8cbfadd28ca0f62620604d3750e018fc78a12"} Nov 04 01:33:26.429921 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.429900 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" event={"ID":"5146a1c5-733e-4454-a89d-1071139316c1","Type":"ContainerStarted","Data":"38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292"} Nov 04 01:33:26.448431 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.448411 5051 scope.go:122] "RemoveContainer" containerID="ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea" Nov 04 01:33:26.448768 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:26.448737 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea\": container with ID starting with ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea not found: ID does not exist" containerID="ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea" Nov 04 01:33:26.448879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.448777 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea"} err="failed to get container status \"ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea\": rpc error: code = NotFound desc = could not find container \"ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea\": container with ID starting with ac6e8dd9433beef2199b993cd386bf4f6ad1d30554ff6e86653c669e0fd377ea not found: ID does not exist" Nov 04 01:33:26.475435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.475335 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:33:26.526145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.517161 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46690fdf-fd41-4366-b213-a60d37e78b45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46690fdf-fd41-4366-b213-a60d37e78b45-plugins-config-volume\") pod \"46690fdf-fd41-4366-b213-a60d37e78b45\" (UID: \"46690fdf-fd41-4366-b213-a60d37e78b45\") " Nov 04 01:33:26.526145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.517356 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46690fdf-fd41-4366-b213-a60d37e78b45-kube-api-access-nxfrj\" (UniqueName: \"kubernetes.io/projected/46690fdf-fd41-4366-b213-a60d37e78b45-kube-api-access-nxfrj\") pod \"46690fdf-fd41-4366-b213-a60d37e78b45\" (UID: \"46690fdf-fd41-4366-b213-a60d37e78b45\") " Nov 04 01:33:26.526145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.518587 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46690fdf-fd41-4366-b213-a60d37e78b45-plugins-config-volume" pod "46690fdf-fd41-4366-b213-a60d37e78b45" (UID: "46690fdf-fd41-4366-b213-a60d37e78b45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:26.588246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.574227 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46690fdf-fd41-4366-b213-a60d37e78b45-kube-api-access-nxfrj" pod "46690fdf-fd41-4366-b213-a60d37e78b45" (UID: "46690fdf-fd41-4366-b213-a60d37e78b45"). InnerVolumeSpecName "kube-api-access-nxfrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:26.594145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.589494 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44"] Nov 04 01:33:26.646145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.641848 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-920689-6789748bf4-nks44"] Nov 04 01:33:26.654143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.648294 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxfrj\" (UniqueName: \"kubernetes.io/projected/46690fdf-fd41-4366-b213-a60d37e78b45-kube-api-access-nxfrj\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:26.654143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:26.648332 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46690fdf-fd41-4366-b213-a60d37e78b45-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:27.250921 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.249343 5051 patch_prober.go:29] interesting pod/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 04 01:33:27.250921 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.249486 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 04 01:33:27.285144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.284096 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f531bf-f087-4ccc-a582-91a0a87e04a7" path="/var/lib/kubelet/pods/77f531bf-f087-4ccc-a582-91a0a87e04a7/volumes" Nov 04 01:33:27.285144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.284968 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:33:27.534205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.519104 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" event={"ID":"46690fdf-fd41-4366-b213-a60d37e78b45","Type":"ContainerDied","Data":"6ad6724d9f9c6197cbbbe8361b9f8ed6b9f19f801e2d5d2ab17706c0ef3c23f0"} Nov 04 01:33:27.534205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.519199 5051 scope.go:122] "RemoveContainer" containerID="4fda571039a8f949c7bf414f33a8cbfadd28ca0f62620604d3750e018fc78a12" Nov 04 01:33:27.534205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.519332 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr" Nov 04 01:33:27.555614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.554980 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" podUID="531cfc6a-8853-4788-8d9a-c05d9a233639" containerName="payload-processing" containerID="cri-o://8362a71bd2c2f7230cc5f75f89ed3bf4461c0aeaea9ab171f65892d85f6ce2d1" gracePeriod=25 Nov 04 01:33:27.584671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.583172 5051 generic.go:356] "Generic (PLEG): container finished" podID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerID="31741d3427938c52444f38c23b59ecaa70ba7bf772c346b844883394423ae8d4" exitCode=0 Nov 04 01:33:27.584671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.584485 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" event={"ID":"6897ee6f-4599-48f0-b7d9-3b6484c955bc","Type":"ContainerDied","Data":"31741d3427938c52444f38c23b59ecaa70ba7bf772c346b844883394423ae8d4"} Nov 04 01:33:27.596686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.594224 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" podUID="bac07c86-7302-4f3e-a71b-ac3adcb85fb0" containerName="payload-pre-processing" containerID="cri-o://784e99b3b418341c3278eb5df09dabe8bfd71ff472db00e4db31cbabc6f142f8" gracePeriod=25 Nov 04 01:33:27.636357 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:33:27.626289 5051 logging.go:55] [core] [Channel #1115 SubChannel #1116]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.171:50052", ServerName: "10.132.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.171:50052: connect: connection refused" Nov 04 01:33:27.636357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.628173 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr"] Nov 04 01:33:27.673884 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.673789 5051 scope.go:122] "RemoveContainer" containerID="5aa5f251ad4a72250ea7ce7c0b936a6ac23f0865ea6ab07554508a7e5114fb6e" Nov 04 01:33:27.765145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:27.754786 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-920689-54f48b969f-52qmr"] Nov 04 01:33:28.332154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.331465 5051 patch_prober.go:29] interesting pod/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 04 01:33:28.332154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.331527 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 04 01:33:28.421194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.420240 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.933499 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-data\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.933627 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-credential-socket\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.933668 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-podinfo\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.933753 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-token\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-token\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.933818 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-socket\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.933865 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-certs\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.933919 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-envoy\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.933987 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-kube-api-access-dlrdl\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-kube-api-access-dlrdl\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.001157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:28.934053 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istiod-ca-cert\") pod \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\" (UID: \"6897ee6f-4599-48f0-b7d9-3b6484c955bc\") " Nov 04 01:33:29.039584 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:33:29.021569 5051 logging.go:55] [core] [Channel #1115 SubChannel #1116]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.171:50052", ServerName: "10.132.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.171:50052: connect: connection refused" Nov 04 01:33:29.064023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.041343 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" podUID="531cfc6a-8853-4788-8d9a-c05d9a233639" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.171:50052\" within 1s: context deadline exceeded" Nov 04 01:33:29.168368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.168319 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:33:29.199146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.193162 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-credential-socket" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:29.485603 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.196637 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-data" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:29.649055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.461800 5051 patch_prober.go:29] interesting pod/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 04 01:33:29.649055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.571397 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:29.710206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.197803 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:29.801149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.403210 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:33:29.801149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.423346 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:29.801149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.788106 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 04 01:33:29.876145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.865282 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-podinfo" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:29.876145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.867733 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded" Nov 04 01:33:29.927204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.913419 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" podUID="3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.205:50052\" within 1s: context deadline exceeded" Nov 04 01:33:29.974534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:29.955438 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-socket" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:30.008224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.008175 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" podUID="a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.204:50052\" within 1s: context deadline exceeded" Nov 04 01:33:30.039344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.039278 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.040569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.040521 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-token" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:30.090215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.090164 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded" Nov 04 01:33:30.169714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.169662 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:33:30.170018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.169994 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.192295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.192209 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istiod-ca-cert" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:30.259446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.259398 5051 patch_prober.go:29] interesting pod/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 04 01:33:30.270239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.260198 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 04 01:33:30.514811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.452001 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.630152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.603663 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:33:30.831649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.831534 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p" podUID="e1f2221e-2cb7-4513-a72c-e6e4156d6b73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.872148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.841000 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-g4dbj" podUID="51487e32-bffb-43e9-a770-720b3d6d383c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.842065 5051 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-7vtl7 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": context deadline exceeded" start-of-body= Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.930594 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-7vtl7" podUID="9747f7d7-39f9-452d-9e9a-dbc9de48c89e" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": context deadline exceeded" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.844202 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s" podUID="56a1d019-a504-4116-929d-33ad787904e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:33:30.865502 5051 logging.go:55] [core] [Channel #1122 SubChannel #1123]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.204:50052", ServerName: "10.132.0.204:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:52062->10.132.0.204:50052: use of closed network connection" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:33:30.888485 5051 logging.go:55] [core] [Channel #1120 SubChannel #1121]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.205:50052", ServerName: "10.132.0.205:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.132.0.2:45680->10.132.0.205:50052: use of closed network connection" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.862298 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.888196 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.889914 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.931469 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:30.948183 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:30.931596 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-certs" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:31.075103 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.058046 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.261523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.257451 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.261523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.257484 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.325144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.323836 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:33:31.325357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.325204 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.434992 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.410291 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-envoy" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:31.434992 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.412625 5051 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.379262277s: [/var/lib/containers/storage/overlay/0b818f70d9cf1486206d6aaf267940313ff7df4ebbeed83e9570645ec21b9984/diff /var/log/pods/openshift-ingress_e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h_eb08d7e7-695c-4005-8101-58b8a376eb7c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Nov 04 01:33:31.434992 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.412833 5051 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.373645783s: [/var/lib/containers/storage/overlay/a6792ce2db2979709c3c07f7890176211f3396702bf2f90d5d2ffccc22015cbf/diff /var/log/pods/e2e-models-e2e-worker-w6-499ddd_simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c_28707278-956e-4312-a887-be2241f79f79/main/0.log]; will not log again for this container unless duration exceeds 2s Nov 04 01:33:31.459507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.459474 5051 patch_prober.go:29] interesting pod/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 04 01:33:31.459783 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.459736 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 04 01:33:31.572195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.531867 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6897ee6f-4599-48f0-b7d9-3b6484c955bc-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.699254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.573147 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.702682 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:31.700829 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531cfc6a_8853_4788_8d9a_c05d9a233639.slice/crio-conmon-8362a71bd2c2f7230cc5f75f89ed3bf4461c0aeaea9ab171f65892d85f6ce2d1.scope\": RecentStats: unable to find data in memory cache]" Nov 04 01:33:31.871542 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.871496 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:33:31.895830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.895790 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-kube-api-access-dlrdl" pod "6897ee6f-4599-48f0-b7d9-3b6484c955bc" (UID: "6897ee6f-4599-48f0-b7d9-3b6484c955bc"). InnerVolumeSpecName "kube-api-access-dlrdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:31.930159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.928439 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrdl\" (UniqueName: \"kubernetes.io/projected/6897ee6f-4599-48f0-b7d9-3b6484c955bc-kube-api-access-dlrdl\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:31.930159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.928538 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.930159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.928619 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:31.935151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.932503 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:33:31.935151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.932597 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:33:31.941322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:31.940514 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46690fdf-fd41-4366-b213-a60d37e78b45" path="/var/lib/kubelet/pods/46690fdf-fd41-4366-b213-a60d37e78b45/volumes" Nov 04 01:33:32.147198 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:32.115465 5051 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.891s" Nov 04 01:33:32.199947 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.199911 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:33:32.240147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.207031 5051 generic.go:356] "Generic (PLEG): container finished" podID="bac07c86-7302-4f3e-a71b-ac3adcb85fb0" containerID="784e99b3b418341c3278eb5df09dabe8bfd71ff472db00e4db31cbabc6f142f8" exitCode=0 Nov 04 01:33:32.274150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.243514 5051 patch_prober.go:29] interesting pod/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 04 01:33:32.283894 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.283851 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 04 01:33:32.286240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.278317 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:33:32.330145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.245031 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:32.432207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.414939 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:32.461146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.434289 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:33:32.566143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.530972 5051 generic.go:356] "Generic (PLEG): container finished" podID="531cfc6a-8853-4788-8d9a-c05d9a233639" containerID="8362a71bd2c2f7230cc5f75f89ed3bf4461c0aeaea9ab171f65892d85f6ce2d1" exitCode=0 Nov 04 01:33:32.566818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.566518 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:33:32.639222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.635696 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531cfc6a-8853-4788-8d9a-c05d9a233639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531cfc6a-8853-4788-8d9a-c05d9a233639-plugins-config-volume\") pod \"531cfc6a-8853-4788-8d9a-c05d9a233639\" (UID: \"531cfc6a-8853-4788-8d9a-c05d9a233639\") " Nov 04 01:33:32.639222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.635831 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531cfc6a-8853-4788-8d9a-c05d9a233639-kube-api-access-jw5z2\" (UniqueName: \"kubernetes.io/projected/531cfc6a-8853-4788-8d9a-c05d9a233639-kube-api-access-jw5z2\") pod \"531cfc6a-8853-4788-8d9a-c05d9a233639\" (UID: \"531cfc6a-8853-4788-8d9a-c05d9a233639\") " Nov 04 01:33:32.713193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.709743 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" Nov 04 01:33:32.742147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.739178 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531cfc6a-8853-4788-8d9a-c05d9a233639-plugins-config-volume" pod "531cfc6a-8853-4788-8d9a-c05d9a233639" (UID: "531cfc6a-8853-4788-8d9a-c05d9a233639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:32.742147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.739370 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" event={"ID":"bac07c86-7302-4f3e-a71b-ac3adcb85fb0","Type":"ContainerDied","Data":"784e99b3b418341c3278eb5df09dabe8bfd71ff472db00e4db31cbabc6f142f8"} Nov 04 01:33:32.742147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.739446 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" event={"ID":"531cfc6a-8853-4788-8d9a-c05d9a233639","Type":"ContainerDied","Data":"8362a71bd2c2f7230cc5f75f89ed3bf4461c0aeaea9ab171f65892d85f6ce2d1"} Nov 04 01:33:32.742147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.739495 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz"] Nov 04 01:33:32.742147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.739583 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:33:32.742147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.739600 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6"] Nov 04 01:33:32.742147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.739665 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:33:32.742147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.739698 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:33:32.781896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.781300 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:33:32.781896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.781508 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p" event={"ID":"6897ee6f-4599-48f0-b7d9-3b6484c955bc","Type":"ContainerDied","Data":"15581bc0722b53c03e5c5a2bcbb463b4e2ba756030dfe2fa4a799a6e2e26f871"} Nov 04 01:33:32.781896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.781632 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:33:32.781896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.781700 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:33:32.898095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.867877 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/531cfc6a-8853-4788-8d9a-c05d9a233639-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:32.962149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.898500 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:33:32.962149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:32.899161 5051 scope.go:122] "RemoveContainer" containerID="784e99b3b418341c3278eb5df09dabe8bfd71ff472db00e4db31cbabc6f142f8" Nov 04 01:33:33.024522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.024471 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:33:33.025158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.025102 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:33:33.025623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.025596 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:33:33.045265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.045218 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:33:33.069968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.069924 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-kube-api-access-4qknb\" (UniqueName: \"kubernetes.io/projected/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-kube-api-access-4qknb\") pod \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\" (UID: \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\") " Nov 04 01:33:33.070455 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.070427 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-plugins-config-volume\") pod \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\" (UID: \"bac07c86-7302-4f3e-a71b-ac3adcb85fb0\") " Nov 04 01:33:33.181670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.181628 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-plugins-config-volume" pod "bac07c86-7302-4f3e-a71b-ac3adcb85fb0" (UID: "bac07c86-7302-4f3e-a71b-ac3adcb85fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:33.200573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.199592 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:33.250198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.250161 5051 patch_prober.go:29] interesting pod/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 04 01:33:33.252227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.252189 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 04 01:33:33.427214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.427096 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531cfc6a-8853-4788-8d9a-c05d9a233639-kube-api-access-jw5z2" pod "531cfc6a-8853-4788-8d9a-c05d9a233639" (UID: "531cfc6a-8853-4788-8d9a-c05d9a233639"). InnerVolumeSpecName "kube-api-access-jw5z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:33.554151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.432007 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:33.554151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.533574 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:33:33.714867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.695768 5051 scope.go:122] "RemoveContainer" containerID="8362a71bd2c2f7230cc5f75f89ed3bf4461c0aeaea9ab171f65892d85f6ce2d1" Nov 04 01:33:33.948198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.946284 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:33:33.948198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.946374 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:33.948198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:33.947760 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw5z2\" (UniqueName: \"kubernetes.io/projected/531cfc6a-8853-4788-8d9a-c05d9a233639-kube-api-access-jw5z2\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.049495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.040307 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl"] Nov 04 01:33:34.075601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.049916 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" containerID="cri-o://38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292" gracePeriod=30 Nov 04 01:33:34.146637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.145693 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:33:34.146637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.145755 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:33:34.155348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.145809 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:34.421881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.395901 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:34.427618 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.422715 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:33:34.442558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.442514 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-kube-api-access-4qknb" pod "bac07c86-7302-4f3e-a71b-ac3adcb85fb0" (UID: "bac07c86-7302-4f3e-a71b-ac3adcb85fb0"). InnerVolumeSpecName "kube-api-access-4qknb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:34.471158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.460461 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qknb\" (UniqueName: \"kubernetes.io/projected/bac07c86-7302-4f3e-a71b-ac3adcb85fb0-kube-api-access-4qknb\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:34.733478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.722408 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:33:34.828187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.765699 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:33:34.828187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.765780 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:34.828187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:34.813548 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:33:35.201993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.200364 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:35.204512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.204474 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:33:35.262311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.240830 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 04 01:33:35.262311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.241286 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-kt9vs" podUID="f80b3f88-24db-4e50-aecc-c1c19664e0ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:35.461516 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.460488 5051 scope.go:122] "RemoveContainer" containerID="31741d3427938c52444f38c23b59ecaa70ba7bf772c346b844883394423ae8d4" Nov 04 01:33:35.501095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.501038 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:35.501329 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.501244 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:33:35.743994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.743947 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:35.872802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.872384 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:33:35.876448 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:35.875592 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:33:35.876448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:35.875826 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:33:36.034197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.000541 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 04 01:33:36.243147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.200602 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" event={"ID":"bac07c86-7302-4f3e-a71b-ac3adcb85fb0","Type":"ContainerDied","Data":"bc735fb73702fbdef1b042193e72f4445da1a167786f366e361818d1789690ff"} Nov 04 01:33:36.243147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.200744 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf" Nov 04 01:33:36.243147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.203227 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:36.251404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.251355 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:36.252869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.252832 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:36.254388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.254353 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" event={"ID":"531cfc6a-8853-4788-8d9a-c05d9a233639","Type":"ContainerDied","Data":"ba974558d575df6e383ee1c3550222813ed621903cf2a72f3925701e6a1ba107"} Nov 04 01:33:36.280751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.257557 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88" Nov 04 01:33:36.642360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.642261 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:33:36.667145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.646360 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": context deadline exceeded" Nov 04 01:33:36.792192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.784500 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 04 01:33:36.882483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:36.881215 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:37.122227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.121639 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:33:37.156144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.147210 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p"] Nov 04 01:33:37.244143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.243266 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:37.254249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.254205 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded" Nov 04 01:33:37.411204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.403721 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:37.546147 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:37.506015 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac07c86_7302_4f3e_a71b_ac3adcb85fb0.slice/crio-bc735fb73702fbdef1b042193e72f4445da1a167786f366e361818d1789690ff\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac07c86_7302_4f3e_a71b_ac3adcb85fb0.slice\": RecentStats: unable to find data in memory cache]" Nov 04 01:33:37.663342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.663241 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:37.667230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.666203 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:33:37.739147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.714504 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-385061-openshift-default-7cb5d4455-ssq8p"] Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.743852 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp"] Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749343 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5b86f87c-1d12-4718-b03f-a32375cf87a5" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749371 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b86f87c-1d12-4718-b03f-a32375cf87a5" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749392 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bac07c86-7302-4f3e-a71b-ac3adcb85fb0" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749400 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac07c86-7302-4f3e-a71b-ac3adcb85fb0" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749416 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3bcb9033-bd09-4223-8655-7d12939824b2" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749424 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bcb9033-bd09-4223-8655-7d12939824b2" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749439 5051 cpu_manager.go:414] "containerMap: removing container" podUID="46690fdf-fd41-4366-b213-a60d37e78b45" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749447 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46690fdf-fd41-4366-b213-a60d37e78b45" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749459 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d92673cc-c7ba-471c-a50d-a9864552e991" containerName="revoke-keys" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749468 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92673cc-c7ba-471c-a50d-a9864552e991" containerName="revoke-keys" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749484 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749491 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749534 5051 cpu_manager.go:414] "containerMap: removing container" podUID="531cfc6a-8853-4788-8d9a-c05d9a233639" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749542 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531cfc6a-8853-4788-8d9a-c05d9a233639" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749555 5051 cpu_manager.go:414] "containerMap: removing container" podUID="77f531bf-f087-4ccc-a582-91a0a87e04a7" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749563 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f531bf-f087-4ccc-a582-91a0a87e04a7" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749583 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3365fd0a-a069-4da8-bd68-b323fb2411ca" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749590 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3365fd0a-a069-4da8-bd68-b323fb2411ca" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749601 5051 cpu_manager.go:414] "containerMap: removing container" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749609 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749617 5051 cpu_manager.go:414] "containerMap: removing container" podUID="17f25302-ecbc-4091-86d9-19234028c089" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749625 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f25302-ecbc-4091-86d9-19234028c089" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749638 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749645 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749664 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749671 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749682 5051 cpu_manager.go:414] "containerMap: removing container" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749689 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749701 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5c8c1b3e-1289-4c31-89d0-e43168ec9619" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749709 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8c1b3e-1289-4c31-89d0-e43168ec9619" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749720 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749730 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749744 5051 cpu_manager.go:414] "containerMap: removing container" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749751 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749767 5051 cpu_manager.go:414] "containerMap: removing container" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749774 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749792 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1530f1ae-087d-4766-aaae-02a5e6531b04" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749800 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1530f1ae-087d-4766-aaae-02a5e6531b04" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749811 5051 cpu_manager.go:414] "containerMap: removing container" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749817 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749835 5051 cpu_manager.go:414] "containerMap: removing container" podUID="af4e2767-65a2-4f53-b23a-673b34dfd0ad" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.749842 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4e2767-65a2-4f53-b23a-673b34dfd0ad" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750010 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f531bf-f087-4ccc-a582-91a0a87e04a7" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750023 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3365fd0a-a069-4da8-bd68-b323fb2411ca" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750032 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a8fc37-9b81-44e3-ae6e-6d1c3b86b020" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750043 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f25302-ecbc-4091-86d9-19234028c089" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750052 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750063 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750072 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3631a39e-5bb5-4de8-9fa2-2830f3f00a94" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750083 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750091 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8c1b3e-1289-4c31-89d0-e43168ec9619" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750100 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9540afba-e788-48cf-9e8e-b8a685b2f8b6" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750110 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="747c4b7d-4c96-42ab-b459-8eb5aaf75e09" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750138 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e34ad9-7fee-457e-9a62-98badb115fb0" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750151 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1530f1ae-087d-4766-aaae-02a5e6531b04" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750161 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="507aa1e3-d0e1-4c14-ba76-3f271e737658" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750171 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4e2767-65a2-4f53-b23a-673b34dfd0ad" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750181 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b86f87c-1d12-4718-b03f-a32375cf87a5" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750193 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac07c86-7302-4f3e-a71b-ac3adcb85fb0" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750204 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bcb9033-bd09-4223-8655-7d12939824b2" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750212 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="46690fdf-fd41-4366-b213-a60d37e78b45" containerName="payload-pre-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750220 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92673cc-c7ba-471c-a50d-a9864552e991" containerName="revoke-keys" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750231 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.750246 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="531cfc6a-8853-4788-8d9a-c05d9a233639" containerName="payload-processing" Nov 04 01:33:37.774147 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:37.770448 5051 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac07c86_7302_4f3e_a71b_ac3adcb85fb0.slice/crio-bc735fb73702fbdef1b042193e72f4445da1a167786f366e361818d1789690ff\": RecentStats: unable to find data in memory cache]" Nov 04 01:33:37.819274 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.818812 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:37.886142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.864678 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp"] Nov 04 01:33:37.886142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.864736 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88"] Nov 04 01:33:37.886142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.864761 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5f572a-54757979db-ddf88"] Nov 04 01:33:37.886142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.864786 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf"] Nov 04 01:33:37.886142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.864803 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5f572a-75b54899b7-tr7vf"] Nov 04 01:33:37.886142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.864899 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:37.932803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.932386 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:33:37.932803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.932506 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:33:37.932803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.932681 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:33:37.932803 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.932737 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:33:37.986897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:37.968918 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:33:38.037823 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.001254 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" podUID="3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" containerName="payload-processing" containerID="cri-o://2adf1bb8b8d4041be74e5fa046cf4359e644ce77ee923fa4a762f56e09f14cd0" gracePeriod=25 Nov 04 01:33:38.087041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.048219 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" podUID="a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" containerName="payload-pre-processing" containerID="cri-o://bd03f89f7b5f51402d35411305b58a7e0e7fdad71a0bc6e483eaa17bd7a25b43" gracePeriod=25 Nov 04 01:33:38.087041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.073608 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgsgk\" (UniqueName: \"kubernetes.io/projected/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-kube-api-access-bgsgk\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.087041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.073784 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.087041 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.073826 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.128944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.122415 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:33:38.285881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.285827 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.286298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.286236 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.287217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.287146 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgsgk\" (UniqueName: \"kubernetes.io/projected/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-kube-api-access-bgsgk\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.360394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.358663 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:33:38.421193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.418486 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:38.660154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.517512 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.660154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.518211 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.749680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.732346 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:38.826325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.751315 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded" Nov 04 01:33:38.861143 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:33:38.856066 5051 logging.go:55] [core] [Channel #1141 SubChannel #1142]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.204:50052", ServerName: "10.132.0.204:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.204:50052: connect: connection refused" Nov 04 01:33:38.897207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:38.884574 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgsgk\" (UniqueName: \"kubernetes.io/projected/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-kube-api-access-bgsgk\") pod \"maas-api-revoke-sub-8c5bf2bd-4cgkp\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:38.897207 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:33:38.888519 5051 logging.go:55] [core] [Channel #1143 SubChannel #1144]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.205:50052", ServerName: "10.132.0.205:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.205:50052: connect: connection refused" Nov 04 01:33:39.022948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.015436 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:39.022948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.018036 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:39.084382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.084345 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:33:39.139448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.139402 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:39.223163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.175680 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:33:39.395272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.359327 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:39.421833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.421775 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:39.454148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.446251 5051 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:39.477689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.477653 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531cfc6a-8853-4788-8d9a-c05d9a233639" path="/var/lib/kubelet/pods/531cfc6a-8853-4788-8d9a-c05d9a233639/volumes" Nov 04 01:33:39.478853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.478825 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" path="/var/lib/kubelet/pods/6897ee6f-4599-48f0-b7d9-3b6484c955bc/volumes" Nov 04 01:33:39.479896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.479874 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac07c86-7302-4f3e-a71b-ac3adcb85fb0" path="/var/lib/kubelet/pods/bac07c86-7302-4f3e-a71b-ac3adcb85fb0/volumes" Nov 04 01:33:39.619727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.619644 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:33:39.624602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.622479 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:33:39.773680 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.773635 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" podUID="a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.204:50052\" within 1s: context deadline exceeded" Nov 04 01:33:39.915248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:39.866273 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" podUID="3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.205:50052\" within 1s: context deadline exceeded" Nov 04 01:33:40.312945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.294953 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 04 01:33:40.312945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.295559 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:33:40.323608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.322420 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:33:40.344239 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.344203 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk"] Nov 04 01:33:40.365766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.365728 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp"] Nov 04 01:33:40.411237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.387528 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:33:40.608349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.608293 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:33:40.650047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.650009 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-kube-api-access-7j2ks\" (UniqueName: \"kubernetes.io/projected/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-kube-api-access-7j2ks\") pod \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\" (UID: \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\") " Nov 04 01:33:40.650346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.650326 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-plugins-config-volume\") pod \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\" (UID: \"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44\") " Nov 04 01:33:40.651628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.651605 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-plugins-config-volume" pod "3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" (UID: "3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:40.651982 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.651958 5051 generic.go:356] "Generic (PLEG): container finished" podID="3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" containerID="2adf1bb8b8d4041be74e5fa046cf4359e644ce77ee923fa4a762f56e09f14cd0" exitCode=0 Nov 04 01:33:40.654275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.652100 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" event={"ID":"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44","Type":"ContainerDied","Data":"2adf1bb8b8d4041be74e5fa046cf4359e644ce77ee923fa4a762f56e09f14cd0"} Nov 04 01:33:40.654457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.654445 5051 scope.go:122] "RemoveContainer" containerID="2adf1bb8b8d4041be74e5fa046cf4359e644ce77ee923fa4a762f56e09f14cd0" Nov 04 01:33:40.678966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.678329 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:33:40.705551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.703384 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-kube-api-access-7j2ks" pod "3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" (UID: "3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44"). InnerVolumeSpecName "kube-api-access-7j2ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:40.738214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.736460 5051 generic.go:356] "Generic (PLEG): container finished" podID="a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" containerID="bd03f89f7b5f51402d35411305b58a7e0e7fdad71a0bc6e483eaa17bd7a25b43" exitCode=0 Nov 04 01:33:40.738214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.736520 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" event={"ID":"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec","Type":"ContainerDied","Data":"bd03f89f7b5f51402d35411305b58a7e0e7fdad71a0bc6e483eaa17bd7a25b43"} Nov 04 01:33:40.738214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.736577 5051 scope.go:122] "RemoveContainer" containerID="bd03f89f7b5f51402d35411305b58a7e0e7fdad71a0bc6e483eaa17bd7a25b43" Nov 04 01:33:40.763319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.757945 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j2ks\" (UniqueName: \"kubernetes.io/projected/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-kube-api-access-7j2ks\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.763319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.757979 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.859897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.859309 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-plugins-config-volume\") pod \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\" (UID: \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\") " Nov 04 01:33:40.859897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.859488 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-kube-api-access-5hrrg\" (UniqueName: \"kubernetes.io/projected/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-kube-api-access-5hrrg\") pod \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\" (UID: \"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec\") " Nov 04 01:33:40.859897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.859773 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-plugins-config-volume" pod "a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" (UID: "a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:40.860109 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.860060 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:40.865279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.865252 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-kube-api-access-5hrrg" pod "a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" (UID: "a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec"). InnerVolumeSpecName "kube-api-access-5hrrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:40.970314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:40.970153 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hrrg\" (UniqueName: \"kubernetes.io/projected/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec-kube-api-access-5hrrg\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:41.749322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.749288 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" event={"ID":"a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec","Type":"ContainerDied","Data":"f6f340e6bf1bd99ae817fd74e48a0d044c4b6a2002dfe2820884864d74a5c6a7"} Nov 04 01:33:41.749322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.749306 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6" Nov 04 01:33:41.752678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.750839 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" event={"ID":"3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44","Type":"ContainerDied","Data":"0c8d4eac04378cfa971f8af3f2a3883e04cec3b8058953253aab3354990e86f8"} Nov 04 01:33:41.752678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.750938 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz" Nov 04 01:33:41.755562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.755539 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" event={"ID":"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6","Type":"ContainerStarted","Data":"b07167e5cef79bf2c049474c33888ec727b719342419a961c8d31ad4abe00ffe"} Nov 04 01:33:41.755693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.755584 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" event={"ID":"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6","Type":"ContainerStarted","Data":"fafa553111292c63aed496f8fe2315bef2707aaac46731e98c63a1edc9bb264c"} Nov 04 01:33:41.842115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.824695 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" podStartSLOduration=6.824677648 podStartE2EDuration="6.824677648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:41.787007424 +0000 UTC m=+2160.758756295" watchObservedRunningTime="2026-11-04 01:33:41.824677648 +0000 UTC m=+2160.796426565" Nov 04 01:33:41.951827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.951785 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz"] Nov 04 01:33:41.998330 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:41.998241 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c75ea1-76b549449b-rfdnz"] Nov 04 01:33:42.123176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:42.122368 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6"] Nov 04 01:33:42.129754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:42.129549 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c75ea1-6c89d4bf4c-l5kz6"] Nov 04 01:33:42.781722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:42.781403 5051 generic.go:356] "Generic (PLEG): container finished" podID="a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6" containerID="b07167e5cef79bf2c049474c33888ec727b719342419a961c8d31ad4abe00ffe" exitCode=0 Nov 04 01:33:42.781722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:42.781519 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" event={"ID":"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6","Type":"ContainerDied","Data":"b07167e5cef79bf2c049474c33888ec727b719342419a961c8d31ad4abe00ffe"} Nov 04 01:33:43.203814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:43.203729 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" path="/var/lib/kubelet/pods/3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44/volumes" Nov 04 01:33:43.204546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:43.204511 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" path="/var/lib/kubelet/pods/a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec/volumes" Nov 04 01:33:44.117313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.117174 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:44.216276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.216245 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-xks-ca\") pod \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " Nov 04 01:33:44.216688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.216508 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-kube-api-access-bgsgk\" (UniqueName: \"kubernetes.io/projected/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-kube-api-access-bgsgk\") pod \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " Nov 04 01:33:44.216688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.216590 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-openshift-ca\") pod \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\" (UID: \"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6\") " Nov 04 01:33:44.217068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.217038 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-openshift-ca" pod "a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6" (UID: "a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:44.217363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.217338 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.218754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.218723 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-kube-api-access-bgsgk" pod "a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6" (UID: "a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6"). InnerVolumeSpecName "kube-api-access-bgsgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:44.218845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.218813 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-xks-ca" pod "a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6" (UID: "a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:44.319391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.319351 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.319391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.319389 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgsgk\" (UniqueName: \"kubernetes.io/projected/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6-kube-api-access-bgsgk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.771763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.768770 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:33:44.820224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.819012 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" event={"ID":"a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6","Type":"ContainerDied","Data":"fafa553111292c63aed496f8fe2315bef2707aaac46731e98c63a1edc9bb264c"} Nov 04 01:33:44.820224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.819075 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fafa553111292c63aed496f8fe2315bef2707aaac46731e98c63a1edc9bb264c" Nov 04 01:33:44.820224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.819217 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp" Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.834738 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5146a1c5-733e-4454-a89d-1071139316c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5146a1c5-733e-4454-a89d-1071139316c1-istio-podinfo\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.834787 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-credential-socket\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.834888 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-socket\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.834929 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-certs\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.835084 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-data\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.835152 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-istio-token\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-istio-token\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.835198 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-envoy\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.835267 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5146a1c5-733e-4454-a89d-1071139316c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5146a1c5-733e-4454-a89d-1071139316c1-istiod-ca-cert\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.835329 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-kube-api-access-f8h9k\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-kube-api-access-f8h9k\") pod \"5146a1c5-733e-4454-a89d-1071139316c1\" (UID: \"5146a1c5-733e-4454-a89d-1071139316c1\") " Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.836886 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-certs" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.839775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.837738 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.840588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.839858 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-socket" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.840588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.840192 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-data" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.840588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.840202 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5146a1c5-733e-4454-a89d-1071139316c1-istio-podinfo" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:44.840765 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.840591 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5146a1c5-733e-4454-a89d-1071139316c1-istiod-ca-cert" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:44.850414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.850375 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-credential-socket" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.856112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.855661 5051 generic.go:356] "Generic (PLEG): container finished" podID="5146a1c5-733e-4454-a89d-1071139316c1" containerID="38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292" exitCode=0 Nov 04 01:33:44.856112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.855710 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" event={"ID":"5146a1c5-733e-4454-a89d-1071139316c1","Type":"ContainerDied","Data":"38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292"} Nov 04 01:33:44.856112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.855764 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" event={"ID":"5146a1c5-733e-4454-a89d-1071139316c1","Type":"ContainerDied","Data":"bda1d3c3eed4f97fc2448224b5435a012712ed1f5fafe341179ea85403b93820"} Nov 04 01:33:44.856112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.855793 5051 scope.go:122] "RemoveContainer" containerID="38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292" Nov 04 01:33:44.856112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.855943 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl" Nov 04 01:33:44.869145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.868798 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-envoy" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:44.880062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.871688 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-istio-token" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:44.880062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.871835 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-kube-api-access-f8h9k" pod "5146a1c5-733e-4454-a89d-1071139316c1" (UID: "5146a1c5-733e-4454-a89d-1071139316c1"). InnerVolumeSpecName "kube-api-access-f8h9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:44.901874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.901844 5051 scope.go:122] "RemoveContainer" containerID="3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3" Nov 04 01:33:44.914396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.914372 5051 scope.go:122] "RemoveContainer" containerID="38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292" Nov 04 01:33:44.914705 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:44.914675 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292\": container with ID starting with 38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292 not found: ID does not exist" containerID="38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292" Nov 04 01:33:44.914796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.914707 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292"} err="failed to get container status \"38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292\": rpc error: code = NotFound desc = could not find container \"38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292\": container with ID starting with 38cf5c3342217eeaefc5c73acc598f0e949c32f4734bbc1389527fe518f0c292 not found: ID does not exist" Nov 04 01:33:44.914796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.914727 5051 scope.go:122] "RemoveContainer" containerID="3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3" Nov 04 01:33:44.915038 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:44.915021 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3\": container with ID starting with 3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3 not found: ID does not exist" containerID="3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3" Nov 04 01:33:44.915104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.915044 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3"} err="failed to get container status \"3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3\": rpc error: code = NotFound desc = could not find container \"3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3\": container with ID starting with 3ef4e14326ccbf543ac155b863878fdce44f389893e0c7dc592305c4d1cf15d3 not found: ID does not exist" Nov 04 01:33:44.939055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.939022 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8h9k\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-kube-api-access-f8h9k\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.939055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.939053 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5146a1c5-733e-4454-a89d-1071139316c1-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.939298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.939066 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.939298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.939075 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.939298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.939084 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.939298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.939093 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5146a1c5-733e-4454-a89d-1071139316c1-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.939298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.939102 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5146a1c5-733e-4454-a89d-1071139316c1-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:44.939298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:44.939110 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5146a1c5-733e-4454-a89d-1071139316c1-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:45.200340 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:45.200305 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl"] Nov 04 01:33:45.207987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:45.207949 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4f8dd0-openshift-default-59dd57b6d5-t9dhl"] Nov 04 01:33:47.200846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:47.200804 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5146a1c5-733e-4454-a89d-1071139316c1" path="/var/lib/kubelet/pods/5146a1c5-733e-4454-a89d-1071139316c1/volumes" Nov 04 01:33:47.681343 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:47.681298 5051 patch_prober.go:29] interesting pod/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:33:47.681619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:47.681371 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:48.724721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.724688 5051 scope.go:122] "RemoveContainer" containerID="1d24e0f38f51239b98eba6e943827a2ef283bb5e3495aa010d6513e9e62f580c" Nov 04 01:33:48.764145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.762295 5051 patch_prober.go:29] interesting pod/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:33:48.764145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.762373 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:33:48.953107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.900710 5051 generic.go:356] "Generic (PLEG): container finished" podID="c5123781-6eea-46f8-8893-175421c25141" containerID="12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb" exitCode=0 Nov 04 01:33:48.953107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.900776 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" event={"ID":"c5123781-6eea-46f8-8893-175421c25141","Type":"ContainerDied","Data":"12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb"} Nov 04 01:33:48.953107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.901326 5051 scope.go:122] "RemoveContainer" containerID="12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb" Nov 04 01:33:48.953107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.920428 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:48.953107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.920501 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:48.953107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:48.920533 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:49.924637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:49.924604 5051 generic.go:356] "Generic (PLEG): container finished" podID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerID="fb797ed90ff8cea237bf18ed4abcd468d115d8689244c0e3ef1f802ff1c896a5" exitCode=0 Nov 04 01:33:49.925038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:49.924732 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" event={"ID":"12db2eea-fca9-45ab-b564-a7289c92fe5d","Type":"ContainerDied","Data":"fb797ed90ff8cea237bf18ed4abcd468d115d8689244c0e3ef1f802ff1c896a5"} Nov 04 01:33:49.925194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:49.925182 5051 scope.go:122] "RemoveContainer" containerID="fb797ed90ff8cea237bf18ed4abcd468d115d8689244c0e3ef1f802ff1c896a5" Nov 04 01:33:49.929288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:49.927231 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:49.929288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:49.927318 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:49.929288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:49.927380 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:33:49.929288 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:49.927665 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" event={"ID":"c5123781-6eea-46f8-8893-175421c25141","Type":"ContainerStarted","Data":"6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75"} Nov 04 01:33:50.192187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:50.187008 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk"] Nov 04 01:33:50.620817 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:50.620784 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:33:50.621003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:50.620974 5051 patch_prober.go:29] interesting pod/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 04 01:33:50.621060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:50.621018 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 04 01:33:50.962201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:50.961162 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerName="istio-proxy" containerID="cri-o://e17c89ce294400b0265665df5c63545de6d56385dd38653c3f1ef2c2e2f7a8fd" gracePeriod=30 Nov 04 01:33:50.962201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:50.961486 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" event={"ID":"12db2eea-fca9-45ab-b564-a7289c92fe5d","Type":"ContainerStarted","Data":"e17c89ce294400b0265665df5c63545de6d56385dd38653c3f1ef2c2e2f7a8fd"} Nov 04 01:33:51.129145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.129034 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb"] Nov 04 01:33:51.130073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.130045 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" Nov 04 01:33:51.130275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.130256 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" Nov 04 01:33:51.130413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.130398 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" containerName="payload-pre-processing" Nov 04 01:33:51.130515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.130502 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" containerName="payload-pre-processing" Nov 04 01:33:51.130620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.130608 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" Nov 04 01:33:51.130713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.130699 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6897ee6f-4599-48f0-b7d9-3b6484c955bc" containerName="istio-proxy" Nov 04 01:33:51.130824 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.130812 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6" containerName="revoke-keys" Nov 04 01:33:51.130914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.130902 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6" containerName="revoke-keys" Nov 04 01:33:51.131033 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.131019 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" containerName="payload-processing" Nov 04 01:33:51.131160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.131145 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" containerName="payload-processing" Nov 04 01:33:51.131487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.131468 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" Nov 04 01:33:51.131610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.131598 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" Nov 04 01:33:51.131711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.131700 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4dee10b-b79d-4ef1-82f7-2ee4a8ac28ec" containerName="payload-pre-processing" Nov 04 01:33:51.131822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.131809 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6" containerName="revoke-keys" Nov 04 01:33:51.131920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.131909 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0c7d0f-4e3c-4af7-a99f-4f13b3236a44" containerName="payload-processing" Nov 04 01:33:51.157923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.155363 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.208560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.208521 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:33:51.208812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.208595 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:33:51.208812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.208707 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:33:51.208812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.208782 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:33:51.218986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.218908 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb"] Nov 04 01:33:51.286284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.269295 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjd7k\" (UniqueName: \"kubernetes.io/projected/0be4d3cd-d337-4e2d-b812-5c52a9007508-kube-api-access-wjd7k\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.286284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.269420 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.286284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.269575 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.404142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.380369 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjd7k\" (UniqueName: \"kubernetes.io/projected/0be4d3cd-d337-4e2d-b812-5c52a9007508-kube-api-access-wjd7k\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.404142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.380474 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.404142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.380595 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.404142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.381919 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.437144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.430276 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-xks-ca\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.462046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.462008 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjd7k\" (UniqueName: \"kubernetes.io/projected/0be4d3cd-d337-4e2d-b812-5c52a9007508-kube-api-access-wjd7k\") pod \"maas-api-revoke-sub-952175fa-lwtkb\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.559171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.549084 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:51.627301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.626582 5051 patch_prober.go:29] interesting pod/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 04 01:33:51.627301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:51.626646 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 04 01:33:52.188970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.188356 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb"] Nov 04 01:33:52.537142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.502996 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89"] Nov 04 01:33:52.593877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.592968 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" containerID="cri-o://df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4" gracePeriod=30 Nov 04 01:33:52.618484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.614196 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd"] Nov 04 01:33:52.628158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.618985 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" Nov 04 01:33:52.628158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.619018 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5146a1c5-733e-4454-a89d-1071139316c1" containerName="istio-proxy" Nov 04 01:33:52.733586 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.733545 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:52.799002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.798731 5051 patch_prober.go:29] interesting pod/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 04 01:33:52.799002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.798788 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 04 01:33:52.907674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.907038 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:52.907674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.907404 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:52.907674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.907479 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbjl6\" (UniqueName: \"kubernetes.io/projected/f0bab7ad-9dad-458b-84fb-e00dcca78644-kube-api-access-tbjl6\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:52.963256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:52.963167 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd"] Nov 04 01:33:53.009438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.009411 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbjl6\" (UniqueName: \"kubernetes.io/projected/f0bab7ad-9dad-458b-84fb-e00dcca78644-kube-api-access-tbjl6\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:53.009698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.009682 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:53.009966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.009952 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:53.010670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.010647 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:53.012214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.012191 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:53.017363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.017333 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbjl6\" (UniqueName: \"kubernetes.io/projected/f0bab7ad-9dad-458b-84fb-e00dcca78644-kube-api-access-tbjl6\") pod \"maas-api-revoke-sub-965a29d2-cj5zd\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:53.086213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.086112 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" event={"ID":"0be4d3cd-d337-4e2d-b812-5c52a9007508","Type":"ContainerStarted","Data":"0a97445506d18008252b1099e8185aa35764de8da07ba03fb0b70f4b1f977077"} Nov 04 01:33:53.086213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.086184 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" event={"ID":"0be4d3cd-d337-4e2d-b812-5c52a9007508","Type":"ContainerStarted","Data":"4437bba0a6f89436ff59e34bccecfb2626dcfa513c98188b5693c60ce118f516"} Nov 04 01:33:53.104772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.104713 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" podStartSLOduration=2.104699791 podStartE2EDuration="2.104699791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:33:53.103952084 +0000 UTC m=+2172.075700960" watchObservedRunningTime="2026-11-04 01:33:53.104699791 +0000 UTC m=+2172.076448656" Nov 04 01:33:53.236865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.236830 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:53.621301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.621267 5051 patch_prober.go:29] interesting pod/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 04 01:33:53.621482 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.621323 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 04 01:33:53.693752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:53.693715 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd"] Nov 04 01:33:53.696091 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:33:53.696060 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bab7ad_9dad_458b_84fb_e00dcca78644.slice/crio-2115feb374fa7693c138efbb28f5274fc2f397a77118a214c78ba68a37a144b9 WatchSource:0}: Error finding container 2115feb374fa7693c138efbb28f5274fc2f397a77118a214c78ba68a37a144b9: Status 404 returned error can't find the container with id 2115feb374fa7693c138efbb28f5274fc2f397a77118a214c78ba68a37a144b9 Nov 04 01:33:54.112186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:54.108019 5051 generic.go:356] "Generic (PLEG): container finished" podID="0be4d3cd-d337-4e2d-b812-5c52a9007508" containerID="0a97445506d18008252b1099e8185aa35764de8da07ba03fb0b70f4b1f977077" exitCode=0 Nov 04 01:33:54.112186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:54.108313 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" event={"ID":"0be4d3cd-d337-4e2d-b812-5c52a9007508","Type":"ContainerDied","Data":"0a97445506d18008252b1099e8185aa35764de8da07ba03fb0b70f4b1f977077"} Nov 04 01:33:54.116195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:54.114162 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" event={"ID":"f0bab7ad-9dad-458b-84fb-e00dcca78644","Type":"ContainerStarted","Data":"2115feb374fa7693c138efbb28f5274fc2f397a77118a214c78ba68a37a144b9"} Nov 04 01:33:54.621271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:54.621178 5051 patch_prober.go:29] interesting pod/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 04 01:33:54.621271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:54.621240 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 04 01:33:55.129626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.129591 5051 generic.go:356] "Generic (PLEG): container finished" podID="f0bab7ad-9dad-458b-84fb-e00dcca78644" containerID="f770bfead0e59e525d9fa704c1447aa8dadbe3683c2d8c9ca7a2a6578a69b0cb" exitCode=0 Nov 04 01:33:55.129806 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.129676 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" event={"ID":"f0bab7ad-9dad-458b-84fb-e00dcca78644","Type":"ContainerDied","Data":"f770bfead0e59e525d9fa704c1447aa8dadbe3683c2d8c9ca7a2a6578a69b0cb"} Nov 04 01:33:55.388224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.388191 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:55.522144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.520766 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be4d3cd-d337-4e2d-b812-5c52a9007508-kube-api-access-wjd7k\" (UniqueName: \"kubernetes.io/projected/0be4d3cd-d337-4e2d-b812-5c52a9007508-kube-api-access-wjd7k\") pod \"0be4d3cd-d337-4e2d-b812-5c52a9007508\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " Nov 04 01:33:55.522144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.521028 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-openshift-ca\") pod \"0be4d3cd-d337-4e2d-b812-5c52a9007508\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " Nov 04 01:33:55.522144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.521078 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-xks-ca\") pod \"0be4d3cd-d337-4e2d-b812-5c52a9007508\" (UID: \"0be4d3cd-d337-4e2d-b812-5c52a9007508\") " Nov 04 01:33:55.523351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.523321 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-openshift-ca" pod "0be4d3cd-d337-4e2d-b812-5c52a9007508" (UID: "0be4d3cd-d337-4e2d-b812-5c52a9007508"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:55.524293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.524270 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-xks-ca" pod "0be4d3cd-d337-4e2d-b812-5c52a9007508" (UID: "0be4d3cd-d337-4e2d-b812-5c52a9007508"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:55.525618 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.525592 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be4d3cd-d337-4e2d-b812-5c52a9007508-kube-api-access-wjd7k" pod "0be4d3cd-d337-4e2d-b812-5c52a9007508" (UID: "0be4d3cd-d337-4e2d-b812-5c52a9007508"). InnerVolumeSpecName "kube-api-access-wjd7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:55.620374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.620333 5051 patch_prober.go:29] interesting pod/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Nov 04 01:33:55.620538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.620405 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Nov 04 01:33:55.624023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.623814 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.624023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.623850 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be4d3cd-d337-4e2d-b812-5c52a9007508-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.624023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.623866 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjd7k\" (UniqueName: \"kubernetes.io/projected/0be4d3cd-d337-4e2d-b812-5c52a9007508-kube-api-access-wjd7k\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:55.642694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:55.642662 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c"] Nov 04 01:33:56.139806 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.139483 5051 generic.go:356] "Generic (PLEG): container finished" podID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerID="e17c89ce294400b0265665df5c63545de6d56385dd38653c3f1ef2c2e2f7a8fd" exitCode=0 Nov 04 01:33:56.139806 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.139558 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" event={"ID":"12db2eea-fca9-45ab-b564-a7289c92fe5d","Type":"ContainerDied","Data":"e17c89ce294400b0265665df5c63545de6d56385dd38653c3f1ef2c2e2f7a8fd"} Nov 04 01:33:56.140248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.140221 5051 scope.go:122] "RemoveContainer" containerID="fb797ed90ff8cea237bf18ed4abcd468d115d8689244c0e3ef1f802ff1c896a5" Nov 04 01:33:56.146084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.145623 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" Nov 04 01:33:56.148245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.148216 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb" event={"ID":"0be4d3cd-d337-4e2d-b812-5c52a9007508","Type":"ContainerDied","Data":"4437bba0a6f89436ff59e34bccecfb2626dcfa513c98188b5693c60ce118f516"} Nov 04 01:33:56.148360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.148272 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4437bba0a6f89436ff59e34bccecfb2626dcfa513c98188b5693c60ce118f516" Nov 04 01:33:56.457185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.457093 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:33:56.470019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.469997 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:56.544169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544115 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-token\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-token\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.544359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544203 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bab7ad-9dad-458b-84fb-e00dcca78644-kube-api-access-tbjl6\" (UniqueName: \"kubernetes.io/projected/f0bab7ad-9dad-458b-84fb-e00dcca78644-kube-api-access-tbjl6\") pod \"f0bab7ad-9dad-458b-84fb-e00dcca78644\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " Nov 04 01:33:56.544359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544267 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12db2eea-fca9-45ab-b564-a7289c92fe5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12db2eea-fca9-45ab-b564-a7289c92fe5d-istiod-ca-cert\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.544359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544316 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-credential-socket\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.544608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544381 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-xks-ca\") pod \"f0bab7ad-9dad-458b-84fb-e00dcca78644\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " Nov 04 01:33:56.544608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544416 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-kube-api-access-pdgnt\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-kube-api-access-pdgnt\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.544608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544476 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-openshift-ca\") pod \"f0bab7ad-9dad-458b-84fb-e00dcca78644\" (UID: \"f0bab7ad-9dad-458b-84fb-e00dcca78644\") " Nov 04 01:33:56.544785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544628 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-podinfo\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.544785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544688 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-certs\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.544785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544732 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-data\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.544785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544768 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-envoy\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.545008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544706 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12db2eea-fca9-45ab-b564-a7289c92fe5d-istiod-ca-cert" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:56.545067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.544991 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-certs" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:56.545067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.545052 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-data" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:56.545591 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.545561 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12db2eea-fca9-45ab-b564-a7289c92fe5d-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.545676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.545595 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.545676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.545610 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.546559 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.546534 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-credential-socket" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:56.546769 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.546744 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-openshift-ca" pod "f0bab7ad-9dad-458b-84fb-e00dcca78644" (UID: "f0bab7ad-9dad-458b-84fb-e00dcca78644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:56.548276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.548238 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-token" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:56.548428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.548384 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-kube-api-access-pdgnt" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "kube-api-access-pdgnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:56.548560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.548543 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-envoy" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:56.548735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.548710 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-podinfo" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:56.548962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.548937 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bab7ad-9dad-458b-84fb-e00dcca78644-kube-api-access-tbjl6" pod "f0bab7ad-9dad-458b-84fb-e00dcca78644" (UID: "f0bab7ad-9dad-458b-84fb-e00dcca78644"). InnerVolumeSpecName "kube-api-access-tbjl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:56.549866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.549846 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-xks-ca" pod "f0bab7ad-9dad-458b-84fb-e00dcca78644" (UID: "f0bab7ad-9dad-458b-84fb-e00dcca78644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:56.625365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.625279 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:33:56.625857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.625824 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:33:56.626714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.626684 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:33:56.646211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.646182 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-socket\") pod \"12db2eea-fca9-45ab-b564-a7289c92fe5d\" (UID: \"12db2eea-fca9-45ab-b564-a7289c92fe5d\") " Nov 04 01:33:56.646459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.646437 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-socket" pod "12db2eea-fca9-45ab-b564-a7289c92fe5d" (UID: "12db2eea-fca9-45ab-b564-a7289c92fe5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:56.647082 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647056 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.647263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647087 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.647263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647104 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.647263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647151 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbjl6\" (UniqueName: \"kubernetes.io/projected/f0bab7ad-9dad-458b-84fb-e00dcca78644-kube-api-access-tbjl6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.647263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647170 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12db2eea-fca9-45ab-b564-a7289c92fe5d-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.647263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647184 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.647263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647197 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdgnt\" (UniqueName: \"kubernetes.io/projected/12db2eea-fca9-45ab-b564-a7289c92fe5d-kube-api-access-pdgnt\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.647263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647215 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bab7ad-9dad-458b-84fb-e00dcca78644-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.647263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.647224 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12db2eea-fca9-45ab-b564-a7289c92fe5d-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:56.809436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:56.809393 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" containerID="cri-o://54bcbdabcb6926bbe327c008340e7f0f384134c49a4db0af7a61296b3a0d804e" gracePeriod=45 Nov 04 01:33:57.157048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.157017 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" Nov 04 01:33:57.157251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.157224 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk" event={"ID":"12db2eea-fca9-45ab-b564-a7289c92fe5d","Type":"ContainerDied","Data":"554a3e5829a80f04d4f4dbb5cd21ffaa2bf98ce89082f67afde7f297c2eaaa26"} Nov 04 01:33:57.157434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.157411 5051 scope.go:122] "RemoveContainer" containerID="e17c89ce294400b0265665df5c63545de6d56385dd38653c3f1ef2c2e2f7a8fd" Nov 04 01:33:57.160419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.159643 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" event={"ID":"f0bab7ad-9dad-458b-84fb-e00dcca78644","Type":"ContainerDied","Data":"2115feb374fa7693c138efbb28f5274fc2f397a77118a214c78ba68a37a144b9"} Nov 04 01:33:57.160419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.159688 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2115feb374fa7693c138efbb28f5274fc2f397a77118a214c78ba68a37a144b9" Nov 04 01:33:57.160419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.159639 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd" Nov 04 01:33:57.161368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.161347 5051 generic.go:356] "Generic (PLEG): container finished" podID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerID="54bcbdabcb6926bbe327c008340e7f0f384134c49a4db0af7a61296b3a0d804e" exitCode=0 Nov 04 01:33:57.161577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.161558 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" event={"ID":"15e4c8bf-0ad3-436a-85cc-a9f31a94458b","Type":"ContainerDied","Data":"54bcbdabcb6926bbe327c008340e7f0f384134c49a4db0af7a61296b3a0d804e"} Nov 04 01:33:57.161660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.161593 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" event={"ID":"15e4c8bf-0ad3-436a-85cc-a9f31a94458b","Type":"ContainerDied","Data":"15f1f1a1f84e3adc1bf34d427a55bd9cd236810e87d67b5839b582077061fee4"} Nov 04 01:33:57.161660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.161612 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f1f1a1f84e3adc1bf34d427a55bd9cd236810e87d67b5839b582077061fee4" Nov 04 01:33:57.178644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.178625 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:33:57.208719 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.208683 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk"] Nov 04 01:33:57.213398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.213373 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c75ea1-openshift-default-7fd5cf5447-6xghk"] Nov 04 01:33:57.281613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.281571 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-dshm\") pod \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " Nov 04 01:33:57.281794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.281705 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tmp-dir\") pod \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " Nov 04 01:33:57.281877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.281805 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-model-cache\") pod \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " Nov 04 01:33:57.281877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.281835 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-kube-api-access-t2csf\" (UniqueName: \"kubernetes.io/projected/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-kube-api-access-t2csf\") pod \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " Nov 04 01:33:57.281877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.281873 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tls-certs\" (UniqueName: \"kubernetes.io/secret/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tls-certs\") pod \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " Nov 04 01:33:57.282046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.281998 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-home\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-home\") pod \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\" (UID: \"15e4c8bf-0ad3-436a-85cc-a9f31a94458b\") " Nov 04 01:33:57.282105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.282046 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tmp-dir" pod "15e4c8bf-0ad3-436a-85cc-a9f31a94458b" (UID: "15e4c8bf-0ad3-436a-85cc-a9f31a94458b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.282349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.282324 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-home" pod "15e4c8bf-0ad3-436a-85cc-a9f31a94458b" (UID: "15e4c8bf-0ad3-436a-85cc-a9f31a94458b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.282500 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.282479 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-model-cache" pod "15e4c8bf-0ad3-436a-85cc-a9f31a94458b" (UID: "15e4c8bf-0ad3-436a-85cc-a9f31a94458b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.282867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.282843 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.282977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.282870 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.282977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.282886 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.284673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.284643 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-kube-api-access-t2csf" pod "15e4c8bf-0ad3-436a-85cc-a9f31a94458b" (UID: "15e4c8bf-0ad3-436a-85cc-a9f31a94458b"). InnerVolumeSpecName "kube-api-access-t2csf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:57.285243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.285216 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tls-certs" pod "15e4c8bf-0ad3-436a-85cc-a9f31a94458b" (UID: "15e4c8bf-0ad3-436a-85cc-a9f31a94458b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:33:57.285243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.285230 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-dshm" pod "15e4c8bf-0ad3-436a-85cc-a9f31a94458b" (UID: "15e4c8bf-0ad3-436a-85cc-a9f31a94458b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:57.384049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.384015 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2csf\" (UniqueName: \"kubernetes.io/projected/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-kube-api-access-t2csf\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.384049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.384047 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:57.384049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:57.384058 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e4c8bf-0ad3-436a-85cc-a9f31a94458b-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.170086 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.170024 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:33:58.173178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.173150 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" event={"ID":"01f13c29-48fe-41ce-ba7f-5bfe6d073eab","Type":"ContainerDied","Data":"df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4"} Nov 04 01:33:58.173301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.173197 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" Nov 04 01:33:58.173301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.173212 5051 scope.go:122] "RemoveContainer" containerID="df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4" Nov 04 01:33:58.173301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.173151 5051 generic.go:356] "Generic (PLEG): container finished" podID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerID="df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4" exitCode=0 Nov 04 01:33:58.173520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.173340 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89" event={"ID":"01f13c29-48fe-41ce-ba7f-5bfe6d073eab","Type":"ContainerDied","Data":"f537ba5cd65820db24f4c03c371297cabdf47b6f5ed38bcd8b9a3096a828a4a8"} Nov 04 01:33:58.173520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.173366 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk" Nov 04 01:33:58.184358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.184297 5051 scope.go:122] "RemoveContainer" containerID="3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93" Nov 04 01:33:58.195814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.195725 5051 scope.go:122] "RemoveContainer" containerID="df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4" Nov 04 01:33:58.196236 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:58.196202 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4\": container with ID starting with df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4 not found: ID does not exist" containerID="df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4" Nov 04 01:33:58.196353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.196237 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4"} err="failed to get container status \"df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4\": rpc error: code = NotFound desc = could not find container \"df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4\": container with ID starting with df72c3322bfecab9b7db7f4d9bd3f33ae6a625b678a75b6768d08b8f3d735fe4 not found: ID does not exist" Nov 04 01:33:58.196353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.196256 5051 scope.go:122] "RemoveContainer" containerID="3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93" Nov 04 01:33:58.196578 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:33:58.196555 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93\": container with ID starting with 3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93 not found: ID does not exist" containerID="3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93" Nov 04 01:33:58.196629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.196586 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93"} err="failed to get container status \"3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93\": rpc error: code = NotFound desc = could not find container \"3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93\": container with ID starting with 3322ec2f5e5f22dfeb7599e44a4cebb9148844d221989a5dbba0e66594ca3c93 not found: ID does not exist" Nov 04 01:33:58.196785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.196771 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-token\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-token\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.196845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.196829 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-podinfo\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.196919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.196903 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-kube-api-access-x8pqw\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-kube-api-access-x8pqw\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.196953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.196937 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-data\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.197014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.196999 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-socket\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.197069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197059 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-credential-socket\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.197165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197094 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-certs\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.197228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197195 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istiod-ca-cert\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.197294 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197231 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-envoy\") pod \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\" (UID: \"01f13c29-48fe-41ce-ba7f-5bfe6d073eab\") " Nov 04 01:33:58.197357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197310 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-socket" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.197421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197380 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-credential-socket" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.197421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197338 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-data" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.197553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197532 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-certs" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.197711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.197691 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istiod-ca-cert" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:33:58.198203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.198178 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.198311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.198208 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.198311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.198225 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.198311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.198239 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.198311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.198252 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.202301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.199816 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-podinfo" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:33:58.202301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.200229 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-kube-api-access-x8pqw" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "kube-api-access-x8pqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:58.202301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.201443 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-token" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:33:58.202301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.202013 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-envoy" pod "01f13c29-48fe-41ce-ba7f-5bfe6d073eab" (UID: "01f13c29-48fe-41ce-ba7f-5bfe6d073eab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:33:58.235371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.232796 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk"] Nov 04 01:33:58.236619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.236589 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-099dd9/distinct-w4-099dd9-kserve-7d564d9dcb-5qsxk"] Nov 04 01:33:58.299821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.299784 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.299821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.299820 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.300084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.299837 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8pqw\" (UniqueName: \"kubernetes.io/projected/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-kube-api-access-x8pqw\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.300084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.299853 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f13c29-48fe-41ce-ba7f-5bfe6d073eab-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:33:58.504063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.504029 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89"] Nov 04 01:33:58.507685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.507630 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-920689-openshift-default-56c4d5f8fc-8wr89"] Nov 04 01:33:58.618357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:58.618323 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt"] Nov 04 01:33:59.078365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:59.078330 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg"] Nov 04 01:33:59.105047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:59.105011 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc"] Nov 04 01:33:59.195492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:59.195449 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" path="/var/lib/kubelet/pods/01f13c29-48fe-41ce-ba7f-5bfe6d073eab/volumes" Nov 04 01:33:59.195985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:59.195925 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" containerID="cri-o://6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75" gracePeriod=30 Nov 04 01:33:59.196705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:59.196270 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" path="/var/lib/kubelet/pods/12db2eea-fca9-45ab-b564-a7289c92fe5d/volumes" Nov 04 01:33:59.196781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:33:59.196766 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" path="/var/lib/kubelet/pods/15e4c8bf-0ad3-436a-85cc-a9f31a94458b/volumes" Nov 04 01:34:04.079001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.078916 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" podUID="8193ef32-0b21-49a2-9494-5d6ad52cdcf2" containerName="payload-processing" containerID="cri-o://101da47d822f52c9fdca6b2707502cd13f9ea0a95feb76fe5f504ae3d94df75b" gracePeriod=25 Nov 04 01:34:04.106378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.106000 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" podUID="352949e5-7e2e-45da-87ab-7350a071772a" containerName="payload-pre-processing" containerID="cri-o://516ae398957db60f1ad6069439ca451491dad585439b1793c2f6ebff5ce63d65" gracePeriod=25 Nov 04 01:34:04.121192 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:04.121115 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:34:04.235221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.235190 5051 generic.go:356] "Generic (PLEG): container finished" podID="352949e5-7e2e-45da-87ab-7350a071772a" containerID="516ae398957db60f1ad6069439ca451491dad585439b1793c2f6ebff5ce63d65" exitCode=0 Nov 04 01:34:04.235350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.235322 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" event={"ID":"352949e5-7e2e-45da-87ab-7350a071772a","Type":"ContainerDied","Data":"516ae398957db60f1ad6069439ca451491dad585439b1793c2f6ebff5ce63d65"} Nov 04 01:34:04.237228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.237205 5051 generic.go:356] "Generic (PLEG): container finished" podID="8193ef32-0b21-49a2-9494-5d6ad52cdcf2" containerID="101da47d822f52c9fdca6b2707502cd13f9ea0a95feb76fe5f504ae3d94df75b" exitCode=0 Nov 04 01:34:04.237326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.237240 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" event={"ID":"8193ef32-0b21-49a2-9494-5d6ad52cdcf2","Type":"ContainerDied","Data":"101da47d822f52c9fdca6b2707502cd13f9ea0a95feb76fe5f504ae3d94df75b"} Nov 04 01:34:04.558684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.558284 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:34:04.625190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.625109 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:34:04.631600 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.631569 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/352949e5-7e2e-45da-87ab-7350a071772a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352949e5-7e2e-45da-87ab-7350a071772a-plugins-config-volume\") pod \"352949e5-7e2e-45da-87ab-7350a071772a\" (UID: \"352949e5-7e2e-45da-87ab-7350a071772a\") " Nov 04 01:34:04.631766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.631744 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352949e5-7e2e-45da-87ab-7350a071772a-kube-api-access-dmk7b\" (UniqueName: \"kubernetes.io/projected/352949e5-7e2e-45da-87ab-7350a071772a-kube-api-access-dmk7b\") pod \"352949e5-7e2e-45da-87ab-7350a071772a\" (UID: \"352949e5-7e2e-45da-87ab-7350a071772a\") " Nov 04 01:34:04.632150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.632096 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352949e5-7e2e-45da-87ab-7350a071772a-plugins-config-volume" pod "352949e5-7e2e-45da-87ab-7350a071772a" (UID: "352949e5-7e2e-45da-87ab-7350a071772a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:04.632484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.632462 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/352949e5-7e2e-45da-87ab-7350a071772a-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.635674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.635639 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352949e5-7e2e-45da-87ab-7350a071772a-kube-api-access-dmk7b" pod "352949e5-7e2e-45da-87ab-7350a071772a" (UID: "352949e5-7e2e-45da-87ab-7350a071772a"). InnerVolumeSpecName "kube-api-access-dmk7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:04.728762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.728730 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:34:04.733021 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.732995 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-kube-api-access-9tww6\") pod \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\" (UID: \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\") " Nov 04 01:34:04.733185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.733116 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-plugins-config-volume\") pod \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\" (UID: \"8193ef32-0b21-49a2-9494-5d6ad52cdcf2\") " Nov 04 01:34:04.733499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.733476 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-plugins-config-volume" pod "8193ef32-0b21-49a2-9494-5d6ad52cdcf2" (UID: "8193ef32-0b21-49a2-9494-5d6ad52cdcf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:04.733557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.733539 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmk7b\" (UniqueName: \"kubernetes.io/projected/352949e5-7e2e-45da-87ab-7350a071772a-kube-api-access-dmk7b\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.735221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.735197 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-kube-api-access-9tww6" pod "8193ef32-0b21-49a2-9494-5d6ad52cdcf2" (UID: "8193ef32-0b21-49a2-9494-5d6ad52cdcf2"). InnerVolumeSpecName "kube-api-access-9tww6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:04.834311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834278 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-socket\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.834493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834338 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-certs\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.834493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834383 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5123781-6eea-46f8-8893-175421c25141-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5123781-6eea-46f8-8893-175421c25141-istiod-ca-cert\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.834493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834475 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-data\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.834644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834553 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-istio-token\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-istio-token\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.834644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834590 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-kube-api-access-7xpbm\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-kube-api-access-7xpbm\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.834644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834611 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-socket" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:04.834784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834709 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-certs" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:04.834784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834729 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5123781-6eea-46f8-8893-175421c25141-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5123781-6eea-46f8-8893-175421c25141-istio-podinfo\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.834902 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.834803 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-data" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:04.835026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.835003 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5123781-6eea-46f8-8893-175421c25141-istiod-ca-cert" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:04.835163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.835143 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-envoy\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.835223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.835204 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-credential-socket\") pod \"c5123781-6eea-46f8-8893-175421c25141\" (UID: \"c5123781-6eea-46f8-8893-175421c25141\") " Nov 04 01:34:04.835517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.835492 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-credential-socket" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:04.836316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.836159 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.836316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.836189 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5123781-6eea-46f8-8893-175421c25141-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.836316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.836223 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.836316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.836240 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tww6\" (UniqueName: \"kubernetes.io/projected/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-kube-api-access-9tww6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.836316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.836257 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8193ef32-0b21-49a2-9494-5d6ad52cdcf2-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.836316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.836272 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.836316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.836288 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.837201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.837180 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5123781-6eea-46f8-8893-175421c25141-istio-podinfo" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:04.837270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.837215 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-kube-api-access-7xpbm" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "kube-api-access-7xpbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:04.837502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.837484 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-envoy" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:04.837629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.837516 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-istio-token" pod "c5123781-6eea-46f8-8893-175421c25141" (UID: "c5123781-6eea-46f8-8893-175421c25141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:04.937448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.937366 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.937448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.937395 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xpbm\" (UniqueName: \"kubernetes.io/projected/c5123781-6eea-46f8-8893-175421c25141-kube-api-access-7xpbm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.937448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.937406 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5123781-6eea-46f8-8893-175421c25141-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:04.937448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:04.937416 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5123781-6eea-46f8-8893-175421c25141-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:05.260922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.260369 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" event={"ID":"8193ef32-0b21-49a2-9494-5d6ad52cdcf2","Type":"ContainerDied","Data":"a93a45fdeae66518c5e8d84b75c0ac9ca35d113bc9f2b5452d6facaa26462395"} Nov 04 01:34:05.260922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.260489 5051 scope.go:122] "RemoveContainer" containerID="101da47d822f52c9fdca6b2707502cd13f9ea0a95feb76fe5f504ae3d94df75b" Nov 04 01:34:05.260922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.260689 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg" Nov 04 01:34:05.265140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.264260 5051 generic.go:356] "Generic (PLEG): container finished" podID="c5123781-6eea-46f8-8893-175421c25141" containerID="6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75" exitCode=0 Nov 04 01:34:05.265140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.264367 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" event={"ID":"c5123781-6eea-46f8-8893-175421c25141","Type":"ContainerDied","Data":"6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75"} Nov 04 01:34:05.265140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.264423 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" event={"ID":"c5123781-6eea-46f8-8893-175421c25141","Type":"ContainerDied","Data":"5f690188e4f67915a00e38eec1ece7662a2eb39e00ba9e9d5d63ca7637a40f6e"} Nov 04 01:34:05.265140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.264547 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt" Nov 04 01:34:05.267944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.267670 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" event={"ID":"352949e5-7e2e-45da-87ab-7350a071772a","Type":"ContainerDied","Data":"de4e1a49e0f216a94fbada218ba16b31a8f3944fcf0a6a5a8a81f00ab126edc4"} Nov 04 01:34:05.267944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.267823 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc" Nov 04 01:34:05.301066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.301039 5051 scope.go:122] "RemoveContainer" containerID="6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75" Nov 04 01:34:05.303858 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.303836 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg"] Nov 04 01:34:05.308026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.307999 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8e9791-5db6f4d97d-xdqpg"] Nov 04 01:34:05.313426 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.313405 5051 scope.go:122] "RemoveContainer" containerID="12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb" Nov 04 01:34:05.317691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.317671 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt"] Nov 04 01:34:05.322573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.322546 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bfc54d-openshift-default-66b5c7b9bc-n28xt"] Nov 04 01:34:05.325093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.325073 5051 scope.go:122] "RemoveContainer" containerID="6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75" Nov 04 01:34:05.325399 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:05.325377 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75\": container with ID starting with 6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75 not found: ID does not exist" containerID="6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75" Nov 04 01:34:05.325490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.325411 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75"} err="failed to get container status \"6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75\": rpc error: code = NotFound desc = could not find container \"6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75\": container with ID starting with 6138e7898d19d7bf12c97b8d6530dee239aafc940915fc4ee59392615fbf1c75 not found: ID does not exist" Nov 04 01:34:05.325490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.325436 5051 scope.go:122] "RemoveContainer" containerID="12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb" Nov 04 01:34:05.325742 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:05.325722 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb\": container with ID starting with 12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb not found: ID does not exist" containerID="12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb" Nov 04 01:34:05.325791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.325750 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb"} err="failed to get container status \"12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb\": rpc error: code = NotFound desc = could not find container \"12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb\": container with ID starting with 12e8a9c73023cbd1b87ae2faa652cdd975a26f0a08229aba86a60b4b9b279bfb not found: ID does not exist" Nov 04 01:34:05.325791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.325767 5051 scope.go:122] "RemoveContainer" containerID="516ae398957db60f1ad6069439ca451491dad585439b1793c2f6ebff5ce63d65" Nov 04 01:34:05.331037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.331013 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc"] Nov 04 01:34:05.337181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.335754 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8e9791-7654dd68d7-ss8pc"] Nov 04 01:34:05.470322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:05.470287 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb"] Nov 04 01:34:07.189863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:07.189830 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352949e5-7e2e-45da-87ab-7350a071772a" path="/var/lib/kubelet/pods/352949e5-7e2e-45da-87ab-7350a071772a/volumes" Nov 04 01:34:07.190480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:07.190460 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8193ef32-0b21-49a2-9494-5d6ad52cdcf2" path="/var/lib/kubelet/pods/8193ef32-0b21-49a2-9494-5d6ad52cdcf2/volumes" Nov 04 01:34:07.191012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:07.190995 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5123781-6eea-46f8-8893-175421c25141" path="/var/lib/kubelet/pods/c5123781-6eea-46f8-8893-175421c25141/volumes" Nov 04 01:34:10.470944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.470892 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" podUID="ba88bc8f-54a5-4b80-a545-343ecbc95551" containerName="payload-processing" containerID="cri-o://6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16" gracePeriod=25 Nov 04 01:34:10.716250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.716195 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" containerID="cri-o://f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8" gracePeriod=45 Nov 04 01:34:10.785461 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.785415 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": dial tcp 10.132.0.64:8000: connect: connection refused" Nov 04 01:34:10.857220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.857184 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:34:10.905566 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.905537 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx"] Nov 04 01:34:10.906397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906377 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8193ef32-0b21-49a2-9494-5d6ad52cdcf2" containerName="payload-processing" Nov 04 01:34:10.906510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906500 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8193ef32-0b21-49a2-9494-5d6ad52cdcf2" containerName="payload-processing" Nov 04 01:34:10.906567 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906560 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" Nov 04 01:34:10.906613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906606 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" Nov 04 01:34:10.906686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906679 5051 cpu_manager.go:414] "containerMap: removing container" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" Nov 04 01:34:10.906737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906731 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" Nov 04 01:34:10.906789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906782 5051 cpu_manager.go:414] "containerMap: removing container" podUID="352949e5-7e2e-45da-87ab-7350a071772a" containerName="payload-pre-processing" Nov 04 01:34:10.906834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906827 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352949e5-7e2e-45da-87ab-7350a071772a" containerName="payload-pre-processing" Nov 04 01:34:10.906900 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906893 5051 cpu_manager.go:414] "containerMap: removing container" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" Nov 04 01:34:10.906951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906944 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" Nov 04 01:34:10.907005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.906999 5051 cpu_manager.go:414] "containerMap: removing container" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerName="istio-proxy" Nov 04 01:34:10.907049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907043 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerName="istio-proxy" Nov 04 01:34:10.907110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907092 5051 cpu_manager.go:414] "containerMap: removing container" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerName="istio-proxy" Nov 04 01:34:10.907210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907199 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerName="istio-proxy" Nov 04 01:34:10.907263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907257 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ba88bc8f-54a5-4b80-a545-343ecbc95551" containerName="payload-processing" Nov 04 01:34:10.907313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907306 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba88bc8f-54a5-4b80-a545-343ecbc95551" containerName="payload-processing" Nov 04 01:34:10.907366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907359 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f0bab7ad-9dad-458b-84fb-e00dcca78644" containerName="revoke-keys" Nov 04 01:34:10.907410 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907403 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bab7ad-9dad-458b-84fb-e00dcca78644" containerName="revoke-keys" Nov 04 01:34:10.907461 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907454 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0be4d3cd-d337-4e2d-b812-5c52a9007508" containerName="revoke-keys" Nov 04 01:34:10.907508 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907501 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be4d3cd-d337-4e2d-b812-5c52a9007508" containerName="revoke-keys" Nov 04 01:34:10.907681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907673 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba88bc8f-54a5-4b80-a545-343ecbc95551" containerName="payload-processing" Nov 04 01:34:10.907750 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907742 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bab7ad-9dad-458b-84fb-e00dcca78644" containerName="revoke-keys" Nov 04 01:34:10.907811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907805 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" Nov 04 01:34:10.907857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907851 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be4d3cd-d337-4e2d-b812-5c52a9007508" containerName="revoke-keys" Nov 04 01:34:10.907907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907901 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8193ef32-0b21-49a2-9494-5d6ad52cdcf2" containerName="payload-processing" Nov 04 01:34:10.907950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907944 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" Nov 04 01:34:10.907995 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.907989 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" Nov 04 01:34:10.908042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.908036 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="352949e5-7e2e-45da-87ab-7350a071772a" containerName="payload-pre-processing" Nov 04 01:34:10.908100 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.908093 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e4c8bf-0ad3-436a-85cc-a9f31a94458b" containerName="main" Nov 04 01:34:10.908163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.908156 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerName="istio-proxy" Nov 04 01:34:10.908219 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.908212 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" Nov 04 01:34:10.908269 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.908263 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="12db2eea-fca9-45ab-b564-a7289c92fe5d" containerName="istio-proxy" Nov 04 01:34:10.911811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.911790 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:10.918378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.918280 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:34:10.918497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.918392 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:34:10.918497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.918452 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:34:10.918640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.918611 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:34:10.927629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.922713 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba88bc8f-54a5-4b80-a545-343ecbc95551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba88bc8f-54a5-4b80-a545-343ecbc95551-plugins-config-volume\") pod \"ba88bc8f-54a5-4b80-a545-343ecbc95551\" (UID: \"ba88bc8f-54a5-4b80-a545-343ecbc95551\") " Nov 04 01:34:10.927629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.922732 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx"] Nov 04 01:34:10.927629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.922913 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba88bc8f-54a5-4b80-a545-343ecbc95551-kube-api-access-pnvcd\" (UniqueName: \"kubernetes.io/projected/ba88bc8f-54a5-4b80-a545-343ecbc95551-kube-api-access-pnvcd\") pod \"ba88bc8f-54a5-4b80-a545-343ecbc95551\" (UID: \"ba88bc8f-54a5-4b80-a545-343ecbc95551\") " Nov 04 01:34:10.927629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.923214 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba88bc8f-54a5-4b80-a545-343ecbc95551-plugins-config-volume" pod "ba88bc8f-54a5-4b80-a545-343ecbc95551" (UID: "ba88bc8f-54a5-4b80-a545-343ecbc95551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:10.927629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.923620 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba88bc8f-54a5-4b80-a545-343ecbc95551-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:10.927629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:10.925607 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba88bc8f-54a5-4b80-a545-343ecbc95551-kube-api-access-pnvcd" pod "ba88bc8f-54a5-4b80-a545-343ecbc95551" (UID: "ba88bc8f-54a5-4b80-a545-343ecbc95551"). InnerVolumeSpecName "kube-api-access-pnvcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:11.024648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.024608 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzkd\" (UniqueName: \"kubernetes.io/projected/30c31cd4-164b-450a-a31b-2f50f99f3e21-kube-api-access-plzkd\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.024831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.024696 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.024831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.024721 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.024831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.024785 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnvcd\" (UniqueName: \"kubernetes.io/projected/ba88bc8f-54a5-4b80-a545-343ecbc95551-kube-api-access-pnvcd\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.064828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.064794 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:34:11.125754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.125722 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152bf905-6686-4d5a-aa60-398bd4125980-kube-api-access-j9gtc\" (UniqueName: \"kubernetes.io/projected/152bf905-6686-4d5a-aa60-398bd4125980-kube-api-access-j9gtc\") pod \"152bf905-6686-4d5a-aa60-398bd4125980\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " Nov 04 01:34:11.125972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.125900 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/152bf905-6686-4d5a-aa60-398bd4125980-tls-certs\" (UniqueName: \"kubernetes.io/secret/152bf905-6686-4d5a-aa60-398bd4125980-tls-certs\") pod \"152bf905-6686-4d5a-aa60-398bd4125980\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " Nov 04 01:34:11.126048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.125996 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-model-cache\") pod \"152bf905-6686-4d5a-aa60-398bd4125980\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " Nov 04 01:34:11.126114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.126046 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-dshm\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-dshm\") pod \"152bf905-6686-4d5a-aa60-398bd4125980\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " Nov 04 01:34:11.126356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.126338 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-home\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-home\") pod \"152bf905-6686-4d5a-aa60-398bd4125980\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " Nov 04 01:34:11.126432 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.126369 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-tmp-dir\") pod \"152bf905-6686-4d5a-aa60-398bd4125980\" (UID: \"152bf905-6686-4d5a-aa60-398bd4125980\") " Nov 04 01:34:11.126833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.126808 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzkd\" (UniqueName: \"kubernetes.io/projected/30c31cd4-164b-450a-a31b-2f50f99f3e21-kube-api-access-plzkd\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.127093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.127069 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-model-cache" pod "152bf905-6686-4d5a-aa60-398bd4125980" (UID: "152bf905-6686-4d5a-aa60-398bd4125980"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.127679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.127566 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.127784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.127741 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-home" pod "152bf905-6686-4d5a-aa60-398bd4125980" (UID: "152bf905-6686-4d5a-aa60-398bd4125980"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.127959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.127925 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-tmp-dir" pod "152bf905-6686-4d5a-aa60-398bd4125980" (UID: "152bf905-6686-4d5a-aa60-398bd4125980"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.128112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.128090 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.129042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.128333 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.129042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.128356 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.129042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.128372 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.129042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.128701 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152bf905-6686-4d5a-aa60-398bd4125980-kube-api-access-j9gtc" pod "152bf905-6686-4d5a-aa60-398bd4125980" (UID: "152bf905-6686-4d5a-aa60-398bd4125980"). InnerVolumeSpecName "kube-api-access-j9gtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:11.129042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.128983 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.129671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.129639 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-dshm" pod "152bf905-6686-4d5a-aa60-398bd4125980" (UID: "152bf905-6686-4d5a-aa60-398bd4125980"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:11.130435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.130413 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.131360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.131337 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152bf905-6686-4d5a-aa60-398bd4125980-tls-certs" pod "152bf905-6686-4d5a-aa60-398bd4125980" (UID: "152bf905-6686-4d5a-aa60-398bd4125980"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:11.135533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.135510 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzkd\" (UniqueName: \"kubernetes.io/projected/30c31cd4-164b-450a-a31b-2f50f99f3e21-kube-api-access-plzkd\") pod \"maas-api-revoke-sub-0eed80c0-h8rvx\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.240548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.232526 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9gtc\" (UniqueName: \"kubernetes.io/projected/152bf905-6686-4d5a-aa60-398bd4125980-kube-api-access-j9gtc\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.240548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.232561 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/152bf905-6686-4d5a-aa60-398bd4125980-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.240548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.232573 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/152bf905-6686-4d5a-aa60-398bd4125980-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:11.240548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.239373 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:11.535554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.535521 5051 generic.go:356] "Generic (PLEG): container finished" podID="ba88bc8f-54a5-4b80-a545-343ecbc95551" containerID="6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16" exitCode=0 Nov 04 01:34:11.535989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.535573 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" Nov 04 01:34:11.535989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.535603 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" event={"ID":"ba88bc8f-54a5-4b80-a545-343ecbc95551","Type":"ContainerDied","Data":"6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16"} Nov 04 01:34:11.535989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.535664 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb" event={"ID":"ba88bc8f-54a5-4b80-a545-343ecbc95551","Type":"ContainerDied","Data":"7edef1a264265bce9649f0967f34e105d4daea8cbe29cfc9acc2c95add4d32eb"} Nov 04 01:34:11.535989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.535698 5051 scope.go:122] "RemoveContainer" containerID="6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16" Nov 04 01:34:11.537497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.537470 5051 generic.go:356] "Generic (PLEG): container finished" podID="152bf905-6686-4d5a-aa60-398bd4125980" containerID="f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8" exitCode=0 Nov 04 01:34:11.537622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.537524 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" Nov 04 01:34:11.537622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.537555 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" event={"ID":"152bf905-6686-4d5a-aa60-398bd4125980","Type":"ContainerDied","Data":"f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8"} Nov 04 01:34:11.537622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.537589 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c" event={"ID":"152bf905-6686-4d5a-aa60-398bd4125980","Type":"ContainerDied","Data":"f6a8354f58910b55b1e7e1f44c9a1455725eb6ecb605d96fd6b626b414866ec9"} Nov 04 01:34:11.546828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.546800 5051 scope.go:122] "RemoveContainer" containerID="6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16" Nov 04 01:34:11.547115 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:11.547092 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16\": container with ID starting with 6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16 not found: ID does not exist" containerID="6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16" Nov 04 01:34:11.547213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.547136 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16"} err="failed to get container status \"6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16\": rpc error: code = NotFound desc = could not find container \"6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16\": container with ID starting with 6a8c8afdbb14d99f00b7d129da0684e6d3ac6d1cece4e0c456ba7c5d1bbf8b16 not found: ID does not exist" Nov 04 01:34:11.547213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.547156 5051 scope.go:122] "RemoveContainer" containerID="f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8" Nov 04 01:34:11.560084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.560060 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx"] Nov 04 01:34:11.560193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.560113 5051 scope.go:122] "RemoveContainer" containerID="f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8" Nov 04 01:34:11.560560 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:11.560533 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8\": container with ID starting with f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8 not found: ID does not exist" containerID="f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8" Nov 04 01:34:11.560645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.560570 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8"} err="failed to get container status \"f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8\": rpc error: code = NotFound desc = could not find container \"f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8\": container with ID starting with f2251b66708e203b87110cbee99b8febea1fb0ed3b030f3100980b6cf4cb6bc8 not found: ID does not exist" Nov 04 01:34:11.565222 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.565180 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c"] Nov 04 01:34:11.569186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.569168 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9c8bf5/simulator-w5-9c8bf5-kserve-6b845fc789-kjm6c"] Nov 04 01:34:11.584457 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.584391 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb"] Nov 04 01:34:11.591169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.591083 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2fff32-79bdfdb756-tvmrb"] Nov 04 01:34:11.828211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.828175 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph"] Nov 04 01:34:11.829032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.829009 5051 cpu_manager.go:414] "containerMap: removing container" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" Nov 04 01:34:11.829032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.829029 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" Nov 04 01:34:11.829283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.829063 5051 cpu_manager.go:414] "containerMap: removing container" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" Nov 04 01:34:11.829283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.829068 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5123781-6eea-46f8-8893-175421c25141" containerName="istio-proxy" Nov 04 01:34:11.829283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.829080 5051 cpu_manager.go:414] "containerMap: removing container" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" Nov 04 01:34:11.829283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.829085 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f13c29-48fe-41ce-ba7f-5bfe6d073eab" containerName="istio-proxy" Nov 04 01:34:11.829283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.829214 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="152bf905-6686-4d5a-aa60-398bd4125980" containerName="main" Nov 04 01:34:11.837084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.835699 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:11.845936 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.845831 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph"] Nov 04 01:34:11.941746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.941717 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdtqz\" (UniqueName: \"kubernetes.io/projected/e350fada-9253-4694-98e8-6fbd8729f39f-kube-api-access-wdtqz\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:11.941906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.941757 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:11.942001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:11.941976 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:12.045147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.043331 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdtqz\" (UniqueName: \"kubernetes.io/projected/e350fada-9253-4694-98e8-6fbd8729f39f-kube-api-access-wdtqz\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:12.045147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.043402 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:12.045147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.043590 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:12.045147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.044829 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:12.076154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.074823 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdtqz\" (UniqueName: \"kubernetes.io/projected/e350fada-9253-4694-98e8-6fbd8729f39f-kube-api-access-wdtqz\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:12.087190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.076510 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d02f17e-vwfph\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:12.167144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.166529 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:12.580630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.580596 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph"] Nov 04 01:34:12.596619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.596584 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" event={"ID":"30c31cd4-164b-450a-a31b-2f50f99f3e21","Type":"ContainerStarted","Data":"7d681f80e448e2d7692d4a7c5f861da2f2df06644b9af26180b3d6b806c14561"} Nov 04 01:34:12.596840 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:12.596819 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" event={"ID":"30c31cd4-164b-450a-a31b-2f50f99f3e21","Type":"ContainerStarted","Data":"8431753c9bfef3ed87d1675b3abec012575531b8b7db91c2153717640b8a4869"} Nov 04 01:34:12.619058 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:34:12.619027 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode350fada_9253_4694_98e8_6fbd8729f39f.slice/crio-30618d908db35c65e26f1d6117b60347143fab3416efe9567d8923f859f94d47 WatchSource:0}: Error finding container 30618d908db35c65e26f1d6117b60347143fab3416efe9567d8923f859f94d47: Status 404 returned error can't find the container with id 30618d908db35c65e26f1d6117b60347143fab3416efe9567d8923f859f94d47 Nov 04 01:34:13.261442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:13.261404 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152bf905-6686-4d5a-aa60-398bd4125980" path="/var/lib/kubelet/pods/152bf905-6686-4d5a-aa60-398bd4125980/volumes" Nov 04 01:34:13.262489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:13.262462 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba88bc8f-54a5-4b80-a545-343ecbc95551" path="/var/lib/kubelet/pods/ba88bc8f-54a5-4b80-a545-343ecbc95551/volumes" Nov 04 01:34:13.608484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:13.608381 5051 generic.go:356] "Generic (PLEG): container finished" podID="30c31cd4-164b-450a-a31b-2f50f99f3e21" containerID="7d681f80e448e2d7692d4a7c5f861da2f2df06644b9af26180b3d6b806c14561" exitCode=0 Nov 04 01:34:13.608484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:13.608465 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" event={"ID":"30c31cd4-164b-450a-a31b-2f50f99f3e21","Type":"ContainerDied","Data":"7d681f80e448e2d7692d4a7c5f861da2f2df06644b9af26180b3d6b806c14561"} Nov 04 01:34:13.610578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:13.610553 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" event={"ID":"e350fada-9253-4694-98e8-6fbd8729f39f","Type":"ContainerStarted","Data":"4bb97f8c68b05e7f8afdef0a3262691abe0c59d164e8d8c4200be55c325f4598"} Nov 04 01:34:13.610791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:13.610755 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" event={"ID":"e350fada-9253-4694-98e8-6fbd8729f39f","Type":"ContainerStarted","Data":"30618d908db35c65e26f1d6117b60347143fab3416efe9567d8923f859f94d47"} Nov 04 01:34:13.639228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:13.639174 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" podStartSLOduration=2.639157174 podStartE2EDuration="2.639157174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:13.635535744 +0000 UTC m=+2192.607284615" watchObservedRunningTime="2026-11-04 01:34:13.639157174 +0000 UTC m=+2192.610906060" Nov 04 01:34:14.258752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.253373 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:14.419142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.399989 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c31cd4-164b-450a-a31b-2f50f99f3e21-kube-api-access-plzkd\" (UniqueName: \"kubernetes.io/projected/30c31cd4-164b-450a-a31b-2f50f99f3e21-kube-api-access-plzkd\") pod \"30c31cd4-164b-450a-a31b-2f50f99f3e21\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " Nov 04 01:34:14.419142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.400170 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-openshift-ca\") pod \"30c31cd4-164b-450a-a31b-2f50f99f3e21\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " Nov 04 01:34:14.419142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.400210 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-xks-ca\") pod \"30c31cd4-164b-450a-a31b-2f50f99f3e21\" (UID: \"30c31cd4-164b-450a-a31b-2f50f99f3e21\") " Nov 04 01:34:14.430846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.430783 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-openshift-ca" pod "30c31cd4-164b-450a-a31b-2f50f99f3e21" (UID: "30c31cd4-164b-450a-a31b-2f50f99f3e21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:14.432731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.431705 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c31cd4-164b-450a-a31b-2f50f99f3e21-kube-api-access-plzkd" pod "30c31cd4-164b-450a-a31b-2f50f99f3e21" (UID: "30c31cd4-164b-450a-a31b-2f50f99f3e21"). InnerVolumeSpecName "kube-api-access-plzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:14.435076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.435036 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-xks-ca" pod "30c31cd4-164b-450a-a31b-2f50f99f3e21" (UID: "30c31cd4-164b-450a-a31b-2f50f99f3e21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:14.507441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.507397 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:14.507441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.507441 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c31cd4-164b-450a-a31b-2f50f99f3e21-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:14.507670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.507456 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plzkd\" (UniqueName: \"kubernetes.io/projected/30c31cd4-164b-450a-a31b-2f50f99f3e21-kube-api-access-plzkd\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:14.621495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.621073 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" Nov 04 01:34:14.621964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.621078 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx" event={"ID":"30c31cd4-164b-450a-a31b-2f50f99f3e21","Type":"ContainerDied","Data":"8431753c9bfef3ed87d1675b3abec012575531b8b7db91c2153717640b8a4869"} Nov 04 01:34:14.621964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.621627 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8431753c9bfef3ed87d1675b3abec012575531b8b7db91c2153717640b8a4869" Nov 04 01:34:14.622934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.622908 5051 generic.go:356] "Generic (PLEG): container finished" podID="e350fada-9253-4694-98e8-6fbd8729f39f" containerID="4bb97f8c68b05e7f8afdef0a3262691abe0c59d164e8d8c4200be55c325f4598" exitCode=0 Nov 04 01:34:14.623052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:14.622995 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" event={"ID":"e350fada-9253-4694-98e8-6fbd8729f39f","Type":"ContainerDied","Data":"4bb97f8c68b05e7f8afdef0a3262691abe0c59d164e8d8c4200be55c325f4598"} Nov 04 01:34:15.534140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.534002 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8"] Nov 04 01:34:15.894161 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.893009 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:15.961507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.961466 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e350fada-9253-4694-98e8-6fbd8729f39f-kube-api-access-wdtqz\" (UniqueName: \"kubernetes.io/projected/e350fada-9253-4694-98e8-6fbd8729f39f-kube-api-access-wdtqz\") pod \"e350fada-9253-4694-98e8-6fbd8729f39f\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " Nov 04 01:34:15.961976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.961947 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-xks-ca\") pod \"e350fada-9253-4694-98e8-6fbd8729f39f\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " Nov 04 01:34:15.962304 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.962278 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-openshift-ca\") pod \"e350fada-9253-4694-98e8-6fbd8729f39f\" (UID: \"e350fada-9253-4694-98e8-6fbd8729f39f\") " Nov 04 01:34:15.962711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.962683 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-openshift-ca" pod "e350fada-9253-4694-98e8-6fbd8729f39f" (UID: "e350fada-9253-4694-98e8-6fbd8729f39f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:15.962903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.962877 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:15.965969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.965939 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e350fada-9253-4694-98e8-6fbd8729f39f-kube-api-access-wdtqz" pod "e350fada-9253-4694-98e8-6fbd8729f39f" (UID: "e350fada-9253-4694-98e8-6fbd8729f39f"). InnerVolumeSpecName "kube-api-access-wdtqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:15.966332 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:15.966305 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-xks-ca" pod "e350fada-9253-4694-98e8-6fbd8729f39f" (UID: "e350fada-9253-4694-98e8-6fbd8729f39f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:16.064717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:16.064642 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdtqz\" (UniqueName: \"kubernetes.io/projected/e350fada-9253-4694-98e8-6fbd8729f39f-kube-api-access-wdtqz\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:16.064717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:16.064672 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e350fada-9253-4694-98e8-6fbd8729f39f-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:16.640735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:16.640700 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" event={"ID":"e350fada-9253-4694-98e8-6fbd8729f39f","Type":"ContainerDied","Data":"30618d908db35c65e26f1d6117b60347143fab3416efe9567d8923f859f94d47"} Nov 04 01:34:16.640735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:16.640720 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph" Nov 04 01:34:16.640976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:16.640748 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30618d908db35c65e26f1d6117b60347143fab3416efe9567d8923f859f94d47" Nov 04 01:34:19.116503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:19.115903 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f"] Nov 04 01:34:19.116503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:19.116155 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" containerID="cri-o://491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450" gracePeriod=30 Nov 04 01:34:19.129087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:19.129053 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp"] Nov 04 01:34:19.129376 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:19.129347 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" containerID="cri-o://828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc" gracePeriod=30 Nov 04 01:34:24.436564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.436215 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.472420 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-credential-socket\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.472584 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-envoy\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.472735 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-podinfo\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.472861 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-certs\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.472909 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-data\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.472955 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-token\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-token\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.472989 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-kube-api-access-z9hp9\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-kube-api-access-z9hp9\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.473050 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istiod-ca-cert\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.474307 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-data" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.474539 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-certs" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.474855 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-socket\") pod \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\" (UID: \"859aa7a6-7b2f-4600-b426-7b89f7416ff0\") " Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.475685 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.475710 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.476050 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-socket" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.476597 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-credential-socket" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.479186 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.478523 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istiod-ca-cert" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:24.496579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.494491 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-podinfo" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:24.496821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.496591 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-kube-api-access-z9hp9" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "kube-api-access-z9hp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.496969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.496866 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-token" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.497159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.497110 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-envoy" pod "859aa7a6-7b2f-4600-b426-7b89f7416ff0" (UID: "859aa7a6-7b2f-4600-b426-7b89f7416ff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.509134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.509097 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:34:24.577042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.577012 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-kube-api-access-kdlth\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-kube-api-access-kdlth\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.577214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.577081 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istiod-ca-cert\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.577214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.577200 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-envoy\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.577544 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.577510 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istiod-ca-cert" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:24.577696 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.577235 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-data\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.577949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.577925 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-data" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.578077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.578049 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-credential-socket" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.578180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.578102 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-credential-socket\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.578249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.578214 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-token\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-token\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.578679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.578655 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-certs\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.579025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.579001 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-certs" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.579113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.579098 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-podinfo\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.579478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.579456 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-socket\") pod \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\" (UID: \"e0a8e42a-646e-46c6-ad74-0b54acadfeae\") " Nov 04 01:34:24.579772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.579738 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-socket" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.580651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.580627 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581188 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581113 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581473 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581612 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581639 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581654 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9hp9\" (UniqueName: \"kubernetes.io/projected/859aa7a6-7b2f-4600-b426-7b89f7416ff0-kube-api-access-z9hp9\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581669 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581684 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859aa7a6-7b2f-4600-b426-7b89f7416ff0-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581698 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581710 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581724 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581737 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859aa7a6-7b2f-4600-b426-7b89f7416ff0-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.581762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.581440 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-token" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.588202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.584491 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-kube-api-access-kdlth" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "kube-api-access-kdlth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:24.588202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.584688 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-envoy" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:24.588202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.587245 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-podinfo" pod "e0a8e42a-646e-46c6-ad74-0b54acadfeae" (UID: "e0a8e42a-646e-46c6-ad74-0b54acadfeae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:24.683309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.683223 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdlth\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-kube-api-access-kdlth\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.683309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.683267 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.683309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.683285 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.683309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.683299 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0a8e42a-646e-46c6-ad74-0b54acadfeae-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:24.715673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.715637 5051 generic.go:356] "Generic (PLEG): container finished" podID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerID="828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc" exitCode=0 Nov 04 01:34:24.715864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.715690 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" Nov 04 01:34:24.715864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.715720 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" event={"ID":"e0a8e42a-646e-46c6-ad74-0b54acadfeae","Type":"ContainerDied","Data":"828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc"} Nov 04 01:34:24.715864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.715799 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp" event={"ID":"e0a8e42a-646e-46c6-ad74-0b54acadfeae","Type":"ContainerDied","Data":"c225eb15a86edfe5322d4c6a7290501cdbdee56271d031b5c3a953c8d4cfe3aa"} Nov 04 01:34:24.715864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.715833 5051 scope.go:122] "RemoveContainer" containerID="828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc" Nov 04 01:34:24.717688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.717663 5051 generic.go:356] "Generic (PLEG): container finished" podID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerID="491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450" exitCode=0 Nov 04 01:34:24.717829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.717699 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" event={"ID":"859aa7a6-7b2f-4600-b426-7b89f7416ff0","Type":"ContainerDied","Data":"491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450"} Nov 04 01:34:24.717829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.717717 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" Nov 04 01:34:24.717829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.717736 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f" event={"ID":"859aa7a6-7b2f-4600-b426-7b89f7416ff0","Type":"ContainerDied","Data":"795125514a92d34e6205af3c4b935507ea8710e780e57de1c259f452ea858f2c"} Nov 04 01:34:24.728881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.728861 5051 scope.go:122] "RemoveContainer" containerID="828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc" Nov 04 01:34:24.729213 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:24.729192 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc\": container with ID starting with 828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc not found: ID does not exist" containerID="828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc" Nov 04 01:34:24.729293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.729221 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc"} err="failed to get container status \"828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc\": rpc error: code = NotFound desc = could not find container \"828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc\": container with ID starting with 828a5a6d0d470fd14624704fa2b700e7aa546283c26db4431b7d9ba619bd9ebc not found: ID does not exist" Nov 04 01:34:24.729293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.729237 5051 scope.go:122] "RemoveContainer" containerID="491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450" Nov 04 01:34:24.738757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.738739 5051 scope.go:122] "RemoveContainer" containerID="b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab" Nov 04 01:34:24.748953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.748935 5051 scope.go:122] "RemoveContainer" containerID="491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450" Nov 04 01:34:24.749281 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:24.749257 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450\": container with ID starting with 491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450 not found: ID does not exist" containerID="491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450" Nov 04 01:34:24.749345 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.749296 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450"} err="failed to get container status \"491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450\": rpc error: code = NotFound desc = could not find container \"491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450\": container with ID starting with 491c6a9e682d66c1264d9d01e9857b7fad0f6914d22f88f31bc3d688212c3450 not found: ID does not exist" Nov 04 01:34:24.749345 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.749320 5051 scope.go:122] "RemoveContainer" containerID="b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab" Nov 04 01:34:24.749648 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:24.749628 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab\": container with ID starting with b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab not found: ID does not exist" containerID="b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab" Nov 04 01:34:24.749760 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.749652 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab"} err="failed to get container status \"b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab\": rpc error: code = NotFound desc = could not find container \"b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab\": container with ID starting with b7d69d688ae74c771fdebdfcce009be892bbb303ce90e5e3a77f96b9a7f7f7ab not found: ID does not exist" Nov 04 01:34:24.765529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.765502 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp"] Nov 04 01:34:24.786097 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.786063 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-gg2dp"] Nov 04 01:34:24.799387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.799361 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f"] Nov 04 01:34:24.810346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:24.810322 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-099dd9-openshift-default-8b7f58cf9-9bv2f"] Nov 04 01:34:25.191556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.191520 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" path="/var/lib/kubelet/pods/859aa7a6-7b2f-4600-b426-7b89f7416ff0/volumes" Nov 04 01:34:25.192310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.192272 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" path="/var/lib/kubelet/pods/e0a8e42a-646e-46c6-ad74-0b54acadfeae/volumes" Nov 04 01:34:25.517983 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.517952 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h"] Nov 04 01:34:25.518724 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518697 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" Nov 04 01:34:25.518826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518725 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" Nov 04 01:34:25.518826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518753 5051 cpu_manager.go:414] "containerMap: removing container" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" Nov 04 01:34:25.518826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518761 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" Nov 04 01:34:25.518826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518810 5051 cpu_manager.go:414] "containerMap: removing container" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" Nov 04 01:34:25.518826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518818 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" Nov 04 01:34:25.519010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518834 5051 cpu_manager.go:414] "containerMap: removing container" podUID="30c31cd4-164b-450a-a31b-2f50f99f3e21" containerName="revoke-keys" Nov 04 01:34:25.519010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518841 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c31cd4-164b-450a-a31b-2f50f99f3e21" containerName="revoke-keys" Nov 04 01:34:25.519010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518861 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e350fada-9253-4694-98e8-6fbd8729f39f" containerName="revoke-keys" Nov 04 01:34:25.519010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518868 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e350fada-9253-4694-98e8-6fbd8729f39f" containerName="revoke-keys" Nov 04 01:34:25.519010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518985 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" Nov 04 01:34:25.519010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.518994 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c31cd4-164b-450a-a31b-2f50f99f3e21" containerName="revoke-keys" Nov 04 01:34:25.519010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.519001 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e350fada-9253-4694-98e8-6fbd8729f39f" containerName="revoke-keys" Nov 04 01:34:25.519010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.519008 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a8e42a-646e-46c6-ad74-0b54acadfeae" containerName="istio-proxy" Nov 04 01:34:25.524198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.524176 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.536174 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.536134 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:34:25.536450 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.536430 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:34:25.536751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.536731 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h"] Nov 04 01:34:25.537295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.537277 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:34:25.537448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.537433 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:34:25.597456 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.597425 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.597617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.597534 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.597689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.597673 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszfn\" (UniqueName: \"kubernetes.io/projected/bc360357-11a4-4403-8d0a-2d07854092e2-kube-api-access-dszfn\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.698788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.698749 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszfn\" (UniqueName: \"kubernetes.io/projected/bc360357-11a4-4403-8d0a-2d07854092e2-kube-api-access-dszfn\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.698964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.698924 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.699022 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.698987 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.699606 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.699578 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.701435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.701412 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.709916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.709882 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszfn\" (UniqueName: \"kubernetes.io/projected/bc360357-11a4-4403-8d0a-2d07854092e2-kube-api-access-dszfn\") pod \"maas-api-revoke-sub-66826164-mrz5h\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:25.847184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:25.840836 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:26.171489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:26.171458 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h"] Nov 04 01:34:26.172073 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:34:26.172034 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc360357_11a4_4403_8d0a_2d07854092e2.slice/crio-cf53f45e85b0c718c910f8ca7a2c15f8f0017250fc1de2b8e2127913fed28417 WatchSource:0}: Error finding container cf53f45e85b0c718c910f8ca7a2c15f8f0017250fc1de2b8e2127913fed28417: Status 404 returned error can't find the container with id cf53f45e85b0c718c910f8ca7a2c15f8f0017250fc1de2b8e2127913fed28417 Nov 04 01:34:26.773201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:26.772466 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" event={"ID":"bc360357-11a4-4403-8d0a-2d07854092e2","Type":"ContainerStarted","Data":"a12780aa7e3ecdc56a9a7667dfd274158673ec900320ed45991ac1a04b347d26"} Nov 04 01:34:26.773201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:26.772533 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" event={"ID":"bc360357-11a4-4403-8d0a-2d07854092e2","Type":"ContainerStarted","Data":"cf53f45e85b0c718c910f8ca7a2c15f8f0017250fc1de2b8e2127913fed28417"} Nov 04 01:34:26.801840 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:26.801783 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" podStartSLOduration=1.801767664 podStartE2EDuration="1.801767664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:26.794509399 +0000 UTC m=+2205.766258271" watchObservedRunningTime="2026-11-04 01:34:26.801767664 +0000 UTC m=+2205.773516580" Nov 04 01:34:27.782479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:27.782441 5051 generic.go:356] "Generic (PLEG): container finished" podID="bc360357-11a4-4403-8d0a-2d07854092e2" containerID="a12780aa7e3ecdc56a9a7667dfd274158673ec900320ed45991ac1a04b347d26" exitCode=0 Nov 04 01:34:27.782998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:27.782525 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" event={"ID":"bc360357-11a4-4403-8d0a-2d07854092e2","Type":"ContainerDied","Data":"a12780aa7e3ecdc56a9a7667dfd274158673ec900320ed45991ac1a04b347d26"} Nov 04 01:34:28.301575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.301532 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn"] Nov 04 01:34:28.302339 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.302311 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="859aa7a6-7b2f-4600-b426-7b89f7416ff0" containerName="istio-proxy" Nov 04 01:34:28.305924 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.305902 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.316618 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.316595 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn"] Nov 04 01:34:28.338844 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.338819 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.338968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.338876 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.339071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.339049 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmlrk\" (UniqueName: \"kubernetes.io/projected/ee166bef-3997-4f6a-a9be-61035433e2e0-kube-api-access-nmlrk\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.440516 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.440481 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.440687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.440572 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.440731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.440705 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmlrk\" (UniqueName: \"kubernetes.io/projected/ee166bef-3997-4f6a-a9be-61035433e2e0-kube-api-access-nmlrk\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.441270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.441249 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.442957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.442930 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.449338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.449313 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmlrk\" (UniqueName: \"kubernetes.io/projected/ee166bef-3997-4f6a-a9be-61035433e2e0-kube-api-access-nmlrk\") pod \"maas-api-revoke-sub-6ca5db96-zqqpn\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.630228 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.630144 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:28.916198 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:34:28.916140 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee166bef_3997_4f6a_a9be_61035433e2e0.slice/crio-2a6f95b9f1e4bc1753565a6cbffd7cf3597616a8405c07217fcf447b1c4a96e5 WatchSource:0}: Error finding container 2a6f95b9f1e4bc1753565a6cbffd7cf3597616a8405c07217fcf447b1c4a96e5: Status 404 returned error can't find the container with id 2a6f95b9f1e4bc1753565a6cbffd7cf3597616a8405c07217fcf447b1c4a96e5 Nov 04 01:34:28.916971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.916945 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn"] Nov 04 01:34:28.982142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:28.982104 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:29.093914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.093628 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-xks-ca\") pod \"bc360357-11a4-4403-8d0a-2d07854092e2\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " Nov 04 01:34:29.093914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.093793 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc360357-11a4-4403-8d0a-2d07854092e2-kube-api-access-dszfn\" (UniqueName: \"kubernetes.io/projected/bc360357-11a4-4403-8d0a-2d07854092e2-kube-api-access-dszfn\") pod \"bc360357-11a4-4403-8d0a-2d07854092e2\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " Nov 04 01:34:29.093914 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.093877 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-openshift-ca\") pod \"bc360357-11a4-4403-8d0a-2d07854092e2\" (UID: \"bc360357-11a4-4403-8d0a-2d07854092e2\") " Nov 04 01:34:29.094300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.094278 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-openshift-ca" pod "bc360357-11a4-4403-8d0a-2d07854092e2" (UID: "bc360357-11a4-4403-8d0a-2d07854092e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:29.094550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.094534 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.150265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.150235 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-xks-ca" pod "bc360357-11a4-4403-8d0a-2d07854092e2" (UID: "bc360357-11a4-4403-8d0a-2d07854092e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:29.150417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.150288 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc360357-11a4-4403-8d0a-2d07854092e2-kube-api-access-dszfn" pod "bc360357-11a4-4403-8d0a-2d07854092e2" (UID: "bc360357-11a4-4403-8d0a-2d07854092e2"). InnerVolumeSpecName "kube-api-access-dszfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:29.195487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.195410 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc360357-11a4-4403-8d0a-2d07854092e2-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.195487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.195436 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dszfn\" (UniqueName: \"kubernetes.io/projected/bc360357-11a4-4403-8d0a-2d07854092e2-kube-api-access-dszfn\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:29.806829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.806788 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" Nov 04 01:34:29.806829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.806803 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h" event={"ID":"bc360357-11a4-4403-8d0a-2d07854092e2","Type":"ContainerDied","Data":"cf53f45e85b0c718c910f8ca7a2c15f8f0017250fc1de2b8e2127913fed28417"} Nov 04 01:34:29.807098 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.806873 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf53f45e85b0c718c910f8ca7a2c15f8f0017250fc1de2b8e2127913fed28417" Nov 04 01:34:29.808869 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.808844 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" event={"ID":"ee166bef-3997-4f6a-a9be-61035433e2e0","Type":"ContainerStarted","Data":"52ba3c040c07f1a05825b49620d3658e4a3bc6bc5a6a062b70f15da07cf6a4cd"} Nov 04 01:34:29.809029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.808899 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" event={"ID":"ee166bef-3997-4f6a-a9be-61035433e2e0","Type":"ContainerStarted","Data":"2a6f95b9f1e4bc1753565a6cbffd7cf3597616a8405c07217fcf447b1c4a96e5"} Nov 04 01:34:29.826212 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:29.826166 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" podStartSLOduration=1.826153053 podStartE2EDuration="1.826153053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:29.823813937 +0000 UTC m=+2208.795562809" watchObservedRunningTime="2026-11-04 01:34:29.826153053 +0000 UTC m=+2208.797901924" Nov 04 01:34:30.564993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:30.564907 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" containerID="cri-o://5fda7a886dbc3cf20238cfd4b0fc315b6b03ddfd46f3ee9f560ba828cce1d26f" gracePeriod=45 Nov 04 01:34:30.818905 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:30.818831 5051 generic.go:356] "Generic (PLEG): container finished" podID="ee166bef-3997-4f6a-a9be-61035433e2e0" containerID="52ba3c040c07f1a05825b49620d3658e4a3bc6bc5a6a062b70f15da07cf6a4cd" exitCode=0 Nov 04 01:34:30.819048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:30.818961 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" event={"ID":"ee166bef-3997-4f6a-a9be-61035433e2e0","Type":"ContainerDied","Data":"52ba3c040c07f1a05825b49620d3658e4a3bc6bc5a6a062b70f15da07cf6a4cd"} Nov 04 01:34:30.821152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:30.821108 5051 generic.go:356] "Generic (PLEG): container finished" podID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerID="5fda7a886dbc3cf20238cfd4b0fc315b6b03ddfd46f3ee9f560ba828cce1d26f" exitCode=0 Nov 04 01:34:30.821331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:30.821181 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" event={"ID":"f1af8632-7d07-4b0a-b1f6-872816217eed","Type":"ContainerDied","Data":"5fda7a886dbc3cf20238cfd4b0fc315b6b03ddfd46f3ee9f560ba828cce1d26f"} Nov 04 01:34:30.950079 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:30.950046 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:34:31.045818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.045781 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1af8632-7d07-4b0a-b1f6-872816217eed-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1af8632-7d07-4b0a-b1f6-872816217eed-tls-certs\") pod \"f1af8632-7d07-4b0a-b1f6-872816217eed\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " Nov 04 01:34:31.046010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.045845 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-model-cache\") pod \"f1af8632-7d07-4b0a-b1f6-872816217eed\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " Nov 04 01:34:31.046010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046002 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1af8632-7d07-4b0a-b1f6-872816217eed-kube-api-access-rxzxz\" (UniqueName: \"kubernetes.io/projected/f1af8632-7d07-4b0a-b1f6-872816217eed-kube-api-access-rxzxz\") pod \"f1af8632-7d07-4b0a-b1f6-872816217eed\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " Nov 04 01:34:31.046265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046197 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-tmp-dir\") pod \"f1af8632-7d07-4b0a-b1f6-872816217eed\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " Nov 04 01:34:31.046265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046239 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-dshm\") pod \"f1af8632-7d07-4b0a-b1f6-872816217eed\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " Nov 04 01:34:31.046265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046265 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-home\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-home\") pod \"f1af8632-7d07-4b0a-b1f6-872816217eed\" (UID: \"f1af8632-7d07-4b0a-b1f6-872816217eed\") " Nov 04 01:34:31.046489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046430 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-tmp-dir" pod "f1af8632-7d07-4b0a-b1f6-872816217eed" (UID: "f1af8632-7d07-4b0a-b1f6-872816217eed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:31.046646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046626 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-home" pod "f1af8632-7d07-4b0a-b1f6-872816217eed" (UID: "f1af8632-7d07-4b0a-b1f6-872816217eed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:31.046835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046804 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-model-cache" pod "f1af8632-7d07-4b0a-b1f6-872816217eed" (UID: "f1af8632-7d07-4b0a-b1f6-872816217eed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:31.046835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046824 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.046835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.046842 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.048392 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.048369 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1af8632-7d07-4b0a-b1f6-872816217eed-tls-certs" pod "f1af8632-7d07-4b0a-b1f6-872816217eed" (UID: "f1af8632-7d07-4b0a-b1f6-872816217eed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:31.048818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.048783 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1af8632-7d07-4b0a-b1f6-872816217eed-kube-api-access-rxzxz" pod "f1af8632-7d07-4b0a-b1f6-872816217eed" (UID: "f1af8632-7d07-4b0a-b1f6-872816217eed"). InnerVolumeSpecName "kube-api-access-rxzxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:31.048906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.048858 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-dshm" pod "f1af8632-7d07-4b0a-b1f6-872816217eed" (UID: "f1af8632-7d07-4b0a-b1f6-872816217eed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:31.148227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.148108 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.148227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.148173 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1af8632-7d07-4b0a-b1f6-872816217eed-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.148227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.148184 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1af8632-7d07-4b0a-b1f6-872816217eed-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.148227 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.148194 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzxz\" (UniqueName: \"kubernetes.io/projected/f1af8632-7d07-4b0a-b1f6-872816217eed-kube-api-access-rxzxz\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:31.831556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.831528 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" Nov 04 01:34:31.832061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.831567 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8" event={"ID":"f1af8632-7d07-4b0a-b1f6-872816217eed","Type":"ContainerDied","Data":"3fbfdc423a2f4e15f48ea89dbc0d5c474ffeccc8a277a176b6fbd96cf4e78f20"} Nov 04 01:34:31.832061 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.831622 5051 scope.go:122] "RemoveContainer" containerID="5fda7a886dbc3cf20238cfd4b0fc315b6b03ddfd46f3ee9f560ba828cce1d26f" Nov 04 01:34:31.859254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.859220 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8"] Nov 04 01:34:31.864645 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:31.864614 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-586541/distinct-2-w1-586541-kserve-66fb97bf5b-hh7j8"] Nov 04 01:34:32.077107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.077081 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:32.173221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.170746 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee166bef-3997-4f6a-a9be-61035433e2e0-kube-api-access-nmlrk\" (UniqueName: \"kubernetes.io/projected/ee166bef-3997-4f6a-a9be-61035433e2e0-kube-api-access-nmlrk\") pod \"ee166bef-3997-4f6a-a9be-61035433e2e0\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " Nov 04 01:34:32.173221 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.170858 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-openshift-ca\") pod \"ee166bef-3997-4f6a-a9be-61035433e2e0\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " Nov 04 01:34:32.191725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.191652 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-openshift-ca" pod "ee166bef-3997-4f6a-a9be-61035433e2e0" (UID: "ee166bef-3997-4f6a-a9be-61035433e2e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:32.194437 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.193369 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee166bef-3997-4f6a-a9be-61035433e2e0-kube-api-access-nmlrk" pod "ee166bef-3997-4f6a-a9be-61035433e2e0" (UID: "ee166bef-3997-4f6a-a9be-61035433e2e0"). InnerVolumeSpecName "kube-api-access-nmlrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:32.276203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.276159 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-xks-ca\") pod \"ee166bef-3997-4f6a-a9be-61035433e2e0\" (UID: \"ee166bef-3997-4f6a-a9be-61035433e2e0\") " Nov 04 01:34:32.276893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.276799 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmlrk\" (UniqueName: \"kubernetes.io/projected/ee166bef-3997-4f6a-a9be-61035433e2e0-kube-api-access-nmlrk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:32.276893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.276825 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:32.278452 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.278427 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-xks-ca" pod "ee166bef-3997-4f6a-a9be-61035433e2e0" (UID: "ee166bef-3997-4f6a-a9be-61035433e2e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:32.378391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.378361 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee166bef-3997-4f6a-a9be-61035433e2e0-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:32.850234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.841178 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" event={"ID":"ee166bef-3997-4f6a-a9be-61035433e2e0","Type":"ContainerDied","Data":"2a6f95b9f1e4bc1753565a6cbffd7cf3597616a8405c07217fcf447b1c4a96e5"} Nov 04 01:34:32.850234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.841244 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6f95b9f1e4bc1753565a6cbffd7cf3597616a8405c07217fcf447b1c4a96e5" Nov 04 01:34:32.850234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:32.841322 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn" Nov 04 01:34:33.026707 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:33.026678 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb"] Nov 04 01:34:33.031300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:33.031275 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2747496-dw9cb"] Nov 04 01:34:33.189596 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:33.189504 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c74efd-d906-43ac-9bea-293d9cde3005" path="/var/lib/kubelet/pods/e5c74efd-d906-43ac-9bea-293d9cde3005/volumes" Nov 04 01:34:33.190644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:33.190622 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" path="/var/lib/kubelet/pods/f1af8632-7d07-4b0a-b1f6-872816217eed/volumes" Nov 04 01:34:33.688014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:33.687971 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57"] Nov 04 01:34:33.688308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:33.688278 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" containerID="cri-o://648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc" gracePeriod=30 Nov 04 01:34:33.695971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:33.695920 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx"] Nov 04 01:34:33.696495 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:33.696457 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" containerID="cri-o://8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d" gracePeriod=30 Nov 04 01:34:34.166395 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:34.166348 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:34:35.901871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:35.901824 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k"] Nov 04 01:34:36.537928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:36.537889 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:36.538230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:36.537953 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:38.526108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:38.526063 5051 patch_prober.go:29] interesting pod/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:34:38.526607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:38.526159 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:34:39.057918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.057885 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:34:39.062575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.062551 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:34:39.122952 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.122911 5051 generic.go:356] "Generic (PLEG): container finished" podID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerID="648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc" exitCode=0 Nov 04 01:34:39.123198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.123016 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" event={"ID":"60599da2-18ec-4ac3-8968-e6726cb5d369","Type":"ContainerDied","Data":"648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc"} Nov 04 01:34:39.123198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.123104 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" event={"ID":"60599da2-18ec-4ac3-8968-e6726cb5d369","Type":"ContainerDied","Data":"fbce760ccb507aced8cb5ee776d9aa2e60bc3b43a837843d11d31ba2e0f2ea13"} Nov 04 01:34:39.123198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.123156 5051 scope.go:122] "RemoveContainer" containerID="648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc" Nov 04 01:34:39.123404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.123324 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57" Nov 04 01:34:39.125362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.125339 5051 generic.go:356] "Generic (PLEG): container finished" podID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerID="8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d" exitCode=0 Nov 04 01:34:39.125759 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.125545 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" event={"ID":"6697cf69-03fa-480f-b8d7-263e7d5d84c7","Type":"ContainerDied","Data":"8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d"} Nov 04 01:34:39.125927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.125912 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" event={"ID":"6697cf69-03fa-480f-b8d7-263e7d5d84c7","Type":"ContainerDied","Data":"2dda14088fd520706aaa5bf248fe5dbf8c1e69a7e13ee77445ec308ab004e9cf"} Nov 04 01:34:39.126047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.125613 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx" Nov 04 01:34:39.137053 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.137034 5051 scope.go:122] "RemoveContainer" containerID="648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc" Nov 04 01:34:39.137365 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:39.137346 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc\": container with ID starting with 648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc not found: ID does not exist" containerID="648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc" Nov 04 01:34:39.137428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.137382 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc"} err="failed to get container status \"648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc\": rpc error: code = NotFound desc = could not find container \"648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc\": container with ID starting with 648fe3b879be14a2dab458eb3529a075ec6746b2dfbb667b018080166d3669fc not found: ID does not exist" Nov 04 01:34:39.137428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.137398 5051 scope.go:122] "RemoveContainer" containerID="8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d" Nov 04 01:34:39.149829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.149808 5051 scope.go:122] "RemoveContainer" containerID="8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d" Nov 04 01:34:39.150140 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:34:39.150093 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d\": container with ID starting with 8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d not found: ID does not exist" containerID="8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d" Nov 04 01:34:39.150238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.150146 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d"} err="failed to get container status \"8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d\": rpc error: code = NotFound desc = could not find container \"8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d\": container with ID starting with 8cc39d6585fbd64677e24712235080f8a36d90ff8aef239f8376e9e7f3a6b86d not found: ID does not exist" Nov 04 01:34:39.156908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.156882 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-certs\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.157030 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.156926 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-envoy\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.157030 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.156987 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-kube-api-access-2fkwq\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-kube-api-access-2fkwq\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.157189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157041 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istiod-ca-cert\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157081 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-podinfo\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157141 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-envoy\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157172 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-token\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-token\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157180 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-certs" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.157462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157227 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-podinfo\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.157462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157278 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-data\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157351 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-credential-socket\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157415 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-kube-api-access-plntc\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-kube-api-access-plntc\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157442 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-data\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.157732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157498 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-socket\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.157732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157585 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-token\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-token\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.157732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157633 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-certs\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157672 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-credential-socket\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.157950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157768 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-socket\") pod \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\" (UID: \"6697cf69-03fa-480f-b8d7-263e7d5d84c7\") " Nov 04 01:34:39.157950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.157808 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60599da2-18ec-4ac3-8968-e6726cb5d369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599da2-18ec-4ac3-8968-e6726cb5d369-istiod-ca-cert\") pod \"60599da2-18ec-4ac3-8968-e6726cb5d369\" (UID: \"60599da2-18ec-4ac3-8968-e6726cb5d369\") " Nov 04 01:34:39.158057 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158002 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-data" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.158311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158171 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istiod-ca-cert" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:39.158417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158311 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-data" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.158729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158520 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-certs" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.158729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158591 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-credential-socket" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.159066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158927 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.159066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158952 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.159066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158956 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-credential-socket" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.159066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.158967 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.159066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.159005 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.159066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.159024 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.159066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.159041 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.159535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.159202 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-socket" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.159535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.159206 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-socket" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.159879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.159849 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60599da2-18ec-4ac3-8968-e6726cb5d369-istiod-ca-cert" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:39.160245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.160207 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-envoy" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.160728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.160611 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-envoy" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:34:39.160728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.160688 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-podinfo" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:39.161575 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.161548 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-token" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.161922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.161891 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-token" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.162208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.162188 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-podinfo" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:34:39.162435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.162404 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-kube-api-access-2fkwq" pod "60599da2-18ec-4ac3-8968-e6726cb5d369" (UID: "60599da2-18ec-4ac3-8968-e6726cb5d369"). InnerVolumeSpecName "kube-api-access-2fkwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.162709 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.162687 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-kube-api-access-plntc" pod "6697cf69-03fa-480f-b8d7-263e7d5d84c7" (UID: "6697cf69-03fa-480f-b8d7-263e7d5d84c7"). InnerVolumeSpecName "kube-api-access-plntc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.260954 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.260994 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plntc\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-kube-api-access-plntc\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261011 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261023 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261037 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261052 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261066 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60599da2-18ec-4ac3-8968-e6726cb5d369-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261080 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60599da2-18ec-4ac3-8968-e6726cb5d369-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261095 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fkwq\" (UniqueName: \"kubernetes.io/projected/60599da2-18ec-4ac3-8968-e6726cb5d369-kube-api-access-2fkwq\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261111 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261139 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.262076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.261153 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6697cf69-03fa-480f-b8d7-263e7d5d84c7-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:39.449636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.449549 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57"] Nov 04 01:34:39.453578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.453547 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-ktr57"] Nov 04 01:34:39.463449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.463416 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx"] Nov 04 01:34:39.468944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:39.468909 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9c8bf5-openshift-default-7f8b8fd6c7-hq4lx"] Nov 04 01:34:40.902761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:40.902723 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" podUID="77c4ea51-856e-4992-bc91-d291fa962765" containerName="payload-processing" containerID="cri-o://513605656f043901892bea4bedf8440265de2d36a3be978237ea975abc5d2c8f" gracePeriod=25 Nov 04 01:34:41.150980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.150694 5051 generic.go:356] "Generic (PLEG): container finished" podID="77c4ea51-856e-4992-bc91-d291fa962765" containerID="513605656f043901892bea4bedf8440265de2d36a3be978237ea975abc5d2c8f" exitCode=0 Nov 04 01:34:41.150980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.150909 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" event={"ID":"77c4ea51-856e-4992-bc91-d291fa962765","Type":"ContainerDied","Data":"513605656f043901892bea4bedf8440265de2d36a3be978237ea975abc5d2c8f"} Nov 04 01:34:41.202642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.202563 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" path="/var/lib/kubelet/pods/60599da2-18ec-4ac3-8968-e6726cb5d369/volumes" Nov 04 01:34:41.203805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.203780 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" path="/var/lib/kubelet/pods/6697cf69-03fa-480f-b8d7-263e7d5d84c7/volumes" Nov 04 01:34:41.246073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.246036 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:34:41.392230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.392183 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c4ea51-856e-4992-bc91-d291fa962765-kube-api-access-f7bcm\" (UniqueName: \"kubernetes.io/projected/77c4ea51-856e-4992-bc91-d291fa962765-kube-api-access-f7bcm\") pod \"77c4ea51-856e-4992-bc91-d291fa962765\" (UID: \"77c4ea51-856e-4992-bc91-d291fa962765\") " Nov 04 01:34:41.392412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.392339 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c4ea51-856e-4992-bc91-d291fa962765-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c4ea51-856e-4992-bc91-d291fa962765-plugins-config-volume\") pod \"77c4ea51-856e-4992-bc91-d291fa962765\" (UID: \"77c4ea51-856e-4992-bc91-d291fa962765\") " Nov 04 01:34:41.392835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.392797 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c4ea51-856e-4992-bc91-d291fa962765-plugins-config-volume" pod "77c4ea51-856e-4992-bc91-d291fa962765" (UID: "77c4ea51-856e-4992-bc91-d291fa962765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:41.393036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.393011 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c4ea51-856e-4992-bc91-d291fa962765-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:41.394886 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.394860 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c4ea51-856e-4992-bc91-d291fa962765-kube-api-access-f7bcm" pod "77c4ea51-856e-4992-bc91-d291fa962765" (UID: "77c4ea51-856e-4992-bc91-d291fa962765"). InnerVolumeSpecName "kube-api-access-f7bcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:41.497173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:41.494859 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bcm\" (UniqueName: \"kubernetes.io/projected/77c4ea51-856e-4992-bc91-d291fa962765-kube-api-access-f7bcm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:42.182175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:42.180014 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" event={"ID":"77c4ea51-856e-4992-bc91-d291fa962765","Type":"ContainerDied","Data":"b62d29a0564cd9043e7bc5637b817d24b179399b29e1583d59f976ea515f7ac7"} Nov 04 01:34:42.182175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:42.180158 5051 scope.go:122] "RemoveContainer" containerID="513605656f043901892bea4bedf8440265de2d36a3be978237ea975abc5d2c8f" Nov 04 01:34:42.182175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:42.180345 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k" Nov 04 01:34:42.241261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:42.240169 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k"] Nov 04 01:34:42.248292 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:42.245168 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-98f6f6-bfc6c9fdb-9zb5k"] Nov 04 01:34:43.217096 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:43.216620 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c4ea51-856e-4992-bc91-d291fa962765" path="/var/lib/kubelet/pods/77c4ea51-856e-4992-bc91-d291fa962765/volumes" Nov 04 01:34:47.700770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.700733 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47"] Nov 04 01:34:47.701317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701303 5051 cpu_manager.go:414] "containerMap: removing container" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" Nov 04 01:34:47.701368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701319 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" Nov 04 01:34:47.701368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701332 5051 cpu_manager.go:414] "containerMap: removing container" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" Nov 04 01:34:47.701368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701337 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" Nov 04 01:34:47.701368 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701365 5051 cpu_manager.go:414] "containerMap: removing container" podUID="77c4ea51-856e-4992-bc91-d291fa962765" containerName="payload-processing" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701372 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c4ea51-856e-4992-bc91-d291fa962765" containerName="payload-processing" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701379 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bc360357-11a4-4403-8d0a-2d07854092e2" containerName="revoke-keys" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701384 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc360357-11a4-4403-8d0a-2d07854092e2" containerName="revoke-keys" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701393 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ee166bef-3997-4f6a-a9be-61035433e2e0" containerName="revoke-keys" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701398 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee166bef-3997-4f6a-a9be-61035433e2e0" containerName="revoke-keys" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701408 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701412 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701492 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c4ea51-856e-4992-bc91-d291fa962765" containerName="payload-processing" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701498 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc360357-11a4-4403-8d0a-2d07854092e2" containerName="revoke-keys" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701507 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee166bef-3997-4f6a-a9be-61035433e2e0" containerName="revoke-keys" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701513 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6697cf69-03fa-480f-b8d7-263e7d5d84c7" containerName="istio-proxy" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701522 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="60599da2-18ec-4ac3-8968-e6726cb5d369" containerName="istio-proxy" Nov 04 01:34:47.701538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.701529 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1af8632-7d07-4b0a-b1f6-872816217eed" containerName="main" Nov 04 01:34:47.706357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.706333 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.710812 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.710780 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:34:47.710981 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.710961 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:34:47.714301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.714274 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47"] Nov 04 01:34:47.714863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.714818 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:34:47.714863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.714844 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:34:47.809945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.809909 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqgx\" (UniqueName: \"kubernetes.io/projected/eef7c944-67e0-4582-93e1-6b2e383e56c0-kube-api-access-jfqgx\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.810213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.809979 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.810213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.810186 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.910979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.910934 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.911207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.911169 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqgx\" (UniqueName: \"kubernetes.io/projected/eef7c944-67e0-4582-93e1-6b2e383e56c0-kube-api-access-jfqgx\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.911318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.911276 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.912039 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.912015 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.913929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.913910 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:47.919562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:47.919535 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqgx\" (UniqueName: \"kubernetes.io/projected/eef7c944-67e0-4582-93e1-6b2e383e56c0-kube-api-access-jfqgx\") pod \"maas-api-revoke-sub-a3c34c34-8fj47\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:48.033412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:48.028619 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:48.335581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:48.335548 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47"] Nov 04 01:34:48.336920 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:34:48.336891 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef7c944_67e0_4582_93e1_6b2e383e56c0.slice/crio-abc4d4b0046d9a385b0fbfb54762df0bdd493e484be19831a3f230d98414ab9c WatchSource:0}: Error finding container abc4d4b0046d9a385b0fbfb54762df0bdd493e484be19831a3f230d98414ab9c: Status 404 returned error can't find the container with id abc4d4b0046d9a385b0fbfb54762df0bdd493e484be19831a3f230d98414ab9c Nov 04 01:34:49.227831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:49.227800 5051 scope.go:122] "RemoveContainer" containerID="54bcbdabcb6926bbe327c008340e7f0f384134c49a4db0af7a61296b3a0d804e" Nov 04 01:34:49.340497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:49.340462 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" event={"ID":"eef7c944-67e0-4582-93e1-6b2e383e56c0","Type":"ContainerStarted","Data":"d58acad9433389a875457796b42e58e9308039161204fd1545380b87025f5e2f"} Nov 04 01:34:49.340612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:49.340521 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" event={"ID":"eef7c944-67e0-4582-93e1-6b2e383e56c0","Type":"ContainerStarted","Data":"abc4d4b0046d9a385b0fbfb54762df0bdd493e484be19831a3f230d98414ab9c"} Nov 04 01:34:49.362821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:49.362682 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" podStartSLOduration=2.362663151 podStartE2EDuration="2.362663151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:34:49.360774231 +0000 UTC m=+2228.332523102" watchObservedRunningTime="2026-11-04 01:34:49.362663151 +0000 UTC m=+2228.334412023" Nov 04 01:34:50.041763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:50.041725 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6"] Nov 04 01:34:50.046968 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:50.046927 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45d7a8cd-8llc6"] Nov 04 01:34:50.348611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:50.348518 5051 generic.go:356] "Generic (PLEG): container finished" podID="eef7c944-67e0-4582-93e1-6b2e383e56c0" containerID="d58acad9433389a875457796b42e58e9308039161204fd1545380b87025f5e2f" exitCode=0 Nov 04 01:34:50.348611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:50.348596 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" event={"ID":"eef7c944-67e0-4582-93e1-6b2e383e56c0","Type":"ContainerDied","Data":"d58acad9433389a875457796b42e58e9308039161204fd1545380b87025f5e2f"} Nov 04 01:34:51.189955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.189914 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f7b38a6-9f62-4105-a347-a9b064d18ce1" path="/var/lib/kubelet/pods/1f7b38a6-9f62-4105-a347-a9b064d18ce1/volumes" Nov 04 01:34:51.531242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.531211 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:51.720785 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.720748 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef7c944-67e0-4582-93e1-6b2e383e56c0-kube-api-access-jfqgx\" (UniqueName: \"kubernetes.io/projected/eef7c944-67e0-4582-93e1-6b2e383e56c0-kube-api-access-jfqgx\") pod \"eef7c944-67e0-4582-93e1-6b2e383e56c0\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " Nov 04 01:34:51.720964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.720905 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-openshift-ca\") pod \"eef7c944-67e0-4582-93e1-6b2e383e56c0\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " Nov 04 01:34:51.721014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.721003 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-xks-ca\") pod \"eef7c944-67e0-4582-93e1-6b2e383e56c0\" (UID: \"eef7c944-67e0-4582-93e1-6b2e383e56c0\") " Nov 04 01:34:51.721350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.721317 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-openshift-ca" pod "eef7c944-67e0-4582-93e1-6b2e383e56c0" (UID: "eef7c944-67e0-4582-93e1-6b2e383e56c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:34:51.722909 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.722883 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef7c944-67e0-4582-93e1-6b2e383e56c0-kube-api-access-jfqgx" pod "eef7c944-67e0-4582-93e1-6b2e383e56c0" (UID: "eef7c944-67e0-4582-93e1-6b2e383e56c0"). InnerVolumeSpecName "kube-api-access-jfqgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:34:51.723019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.722998 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-xks-ca" pod "eef7c944-67e0-4582-93e1-6b2e383e56c0" (UID: "eef7c944-67e0-4582-93e1-6b2e383e56c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:34:51.822926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.822830 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:51.822926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.822871 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfqgx\" (UniqueName: \"kubernetes.io/projected/eef7c944-67e0-4582-93e1-6b2e383e56c0-kube-api-access-jfqgx\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:51.822926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:51.822893 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef7c944-67e0-4582-93e1-6b2e383e56c0-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:34:52.372369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:52.372334 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" Nov 04 01:34:52.372579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:52.372451 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47" event={"ID":"eef7c944-67e0-4582-93e1-6b2e383e56c0","Type":"ContainerDied","Data":"abc4d4b0046d9a385b0fbfb54762df0bdd493e484be19831a3f230d98414ab9c"} Nov 04 01:34:52.372579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:52.372505 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abc4d4b0046d9a385b0fbfb54762df0bdd493e484be19831a3f230d98414ab9c" Nov 04 01:34:52.374616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:52.374583 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd"] Nov 04 01:34:52.378861 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:52.378834 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k"] Nov 04 01:34:55.454965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:55.454925 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl"] Nov 04 01:34:56.152223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:56.152186 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9"] Nov 04 01:34:56.152865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:56.152825 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" containerID="cri-o://7da8a305e495bb78e08f160c2152fa1142422fa8be22b6f95719f9f17cc8e3cf" gracePeriod=30 Nov 04 01:34:56.245780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:56.245439 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h"] Nov 04 01:34:56.246023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:34:56.245991 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" containerID="cri-o://28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30" gracePeriod=30 Nov 04 01:35:00.448642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.448607 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7"] Nov 04 01:35:00.473142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.449170 5051 cpu_manager.go:414] "containerMap: removing container" podUID="eef7c944-67e0-4582-93e1-6b2e383e56c0" containerName="revoke-keys" Nov 04 01:35:00.473142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.449195 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef7c944-67e0-4582-93e1-6b2e383e56c0" containerName="revoke-keys" Nov 04 01:35:00.473142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.449376 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef7c944-67e0-4582-93e1-6b2e383e56c0" containerName="revoke-keys" Nov 04 01:35:00.477255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.477189 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" podUID="7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" containerName="payload-processing" containerID="cri-o://b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e" gracePeriod=25 Nov 04 01:35:00.498615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.481026 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.498615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.492031 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7"] Nov 04 01:35:00.498615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.493018 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:35:00.498615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.493257 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:35:00.498615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.493312 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:00.498615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.493894 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:35:00.532296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.532269 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.532436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.532308 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhll4\" (UniqueName: \"kubernetes.io/projected/b7ecc416-ae43-4785-a349-eae11939a796-kube-api-access-nhll4\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.532545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.532519 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.633345 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.633312 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.634231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.633957 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhll4\" (UniqueName: \"kubernetes.io/projected/b7ecc416-ae43-4785-a349-eae11939a796-kube-api-access-nhll4\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.634231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.633999 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.634231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.634195 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.637539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.637514 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.643034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.643016 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhll4\" (UniqueName: \"kubernetes.io/projected/b7ecc416-ae43-4785-a349-eae11939a796-kube-api-access-nhll4\") pod \"maas-api-revoke-sub-97b0e6ba-dlkx7\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.893319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.893282 5051 patch_prober.go:29] interesting pod/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:35:00.893480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.893357 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:35:00.904105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.904069 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:35:00.919077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.919043 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:00.939975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.939947 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-plugins-config-volume\") pod \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\" (UID: \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\") " Nov 04 01:35:00.940590 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.940481 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-plugins-config-volume" pod "7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" (UID: "7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:00.940955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.940937 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-kube-api-access-bzwxg\" (UniqueName: \"kubernetes.io/projected/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-kube-api-access-bzwxg\") pod \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\" (UID: \"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d\") " Nov 04 01:35:00.941791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.941774 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:00.972109 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:00.971808 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-kube-api-access-bzwxg" pod "7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" (UID: "7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d"). InnerVolumeSpecName "kube-api-access-bzwxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:01.043739 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.043695 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzwxg\" (UniqueName: \"kubernetes.io/projected/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d-kube-api-access-bzwxg\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:01.290895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.288767 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7"] Nov 04 01:35:01.548091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.548061 5051 generic.go:356] "Generic (PLEG): container finished" podID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerID="7da8a305e495bb78e08f160c2152fa1142422fa8be22b6f95719f9f17cc8e3cf" exitCode=0 Nov 04 01:35:01.548571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.548394 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" event={"ID":"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30","Type":"ContainerDied","Data":"7da8a305e495bb78e08f160c2152fa1142422fa8be22b6f95719f9f17cc8e3cf"} Nov 04 01:35:01.550404 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.550373 5051 generic.go:356] "Generic (PLEG): container finished" podID="7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" containerID="b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e" exitCode=0 Nov 04 01:35:01.550695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.550673 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" event={"ID":"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d","Type":"ContainerDied","Data":"b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e"} Nov 04 01:35:01.550866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.550849 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" event={"ID":"7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d","Type":"ContainerDied","Data":"638117b962b243eb501681612d2582265d4cc0cde937d4e622b0e2927004da6c"} Nov 04 01:35:01.550989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.550974 5051 scope.go:122] "RemoveContainer" containerID="b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e" Nov 04 01:35:01.551295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.551274 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl" Nov 04 01:35:01.612930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.612414 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" event={"ID":"b7ecc416-ae43-4785-a349-eae11939a796","Type":"ContainerStarted","Data":"d629579fe9fade688ea442c038e51dfcb7f9a79e8c0cbf3003b02976ab80f23d"} Nov 04 01:35:01.622211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.618225 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl"] Nov 04 01:35:01.622211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.620970 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-ef6cc9-678f897c98-l8fjl"] Nov 04 01:35:01.626747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.626576 5051 scope.go:122] "RemoveContainer" containerID="b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e" Nov 04 01:35:01.627010 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:35:01.626979 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e\": container with ID starting with b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e not found: ID does not exist" containerID="b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e" Nov 04 01:35:01.627260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.627221 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e"} err="failed to get container status \"b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e\": rpc error: code = NotFound desc = could not find container \"b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e\": container with ID starting with b286ca7f54c2fed909c05721d55ccc92039284a55d9594500cd7661bb4665c7e not found: ID does not exist" Nov 04 01:35:01.784873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.784842 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:35:01.947018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.946981 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istiod-ca-cert\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.947342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.947319 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-socket\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.948604 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.948581 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-envoy\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.949499 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.949477 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-certs\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.949629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.949613 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-token\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-token\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.949859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.949841 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-kube-api-access-jjddn\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-kube-api-access-jjddn\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.949997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.949980 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-data\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.950159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.950103 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-podinfo\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.950337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.950317 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-credential-socket\") pod \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\" (UID: \"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30\") " Nov 04 01:35:01.951250 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.951228 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-credential-socket" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.951615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.951595 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-certs" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.956458 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.956280 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-data" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.957263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.957164 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istiod-ca-cert" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:01.957522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.957473 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-socket" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.961849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.961811 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-podinfo" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:01.971981 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.971671 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-kube-api-access-jjddn" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "kube-api-access-jjddn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:01.996825 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.996774 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-envoy" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:01.999152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:01.999099 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-token" pod "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" (UID: "e84c9996-9944-48ca-8ca4-e0ac9d1f8c30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.054946 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.054984 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.055002 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.055017 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjddn\" (UniqueName: \"kubernetes.io/projected/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-kube-api-access-jjddn\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.055034 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.055047 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.055063 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.055078 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.055229 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.055092 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.296350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.296309 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.342560 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-envoy\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.342646 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-token\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-token\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.342713 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-socket\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.342747 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-certs\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.342868 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-credential-socket\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.342901 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-podinfo\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.342964 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb08d7e7-695c-4005-8101-58b8a376eb7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb08d7e7-695c-4005-8101-58b8a376eb7c-istiod-ca-cert\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.343022 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-kube-api-access-72n6d\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-kube-api-access-72n6d\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.348202 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.343064 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-data\") pod \"eb08d7e7-695c-4005-8101-58b8a376eb7c\" (UID: \"eb08d7e7-695c-4005-8101-58b8a376eb7c\") " Nov 04 01:35:02.352984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.352589 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-data" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.360686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.354647 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-certs" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.360686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.354941 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb08d7e7-695c-4005-8101-58b8a376eb7c-istiod-ca-cert" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:02.360686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.354979 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-socket" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.360686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.359557 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-credential-socket" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.374043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.373983 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-podinfo" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:02.374043 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.374005 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-kube-api-access-72n6d" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "kube-api-access-72n6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:02.374317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.374150 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-envoy" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:02.375712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.375680 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-token" pod "eb08d7e7-695c-4005-8101-58b8a376eb7c" (UID: "eb08d7e7-695c-4005-8101-58b8a376eb7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:02.454473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454433 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.454473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454476 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.454702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454495 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.454702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454511 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.454702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454526 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.454702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454542 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.454702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454556 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb08d7e7-695c-4005-8101-58b8a376eb7c-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.454702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454571 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72n6d\" (UniqueName: \"kubernetes.io/projected/eb08d7e7-695c-4005-8101-58b8a376eb7c-kube-api-access-72n6d\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.454702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.454587 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb08d7e7-695c-4005-8101-58b8a376eb7c-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:02.648220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.645689 5051 generic.go:356] "Generic (PLEG): container finished" podID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerID="28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30" exitCode=0 Nov 04 01:35:02.648220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.645858 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" event={"ID":"eb08d7e7-695c-4005-8101-58b8a376eb7c","Type":"ContainerDied","Data":"28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30"} Nov 04 01:35:02.648220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.645915 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" event={"ID":"eb08d7e7-695c-4005-8101-58b8a376eb7c","Type":"ContainerDied","Data":"533d45946591f2a510ecc12c62d9afc01013b87eb9060c18a242f77521bd446a"} Nov 04 01:35:02.648220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.645950 5051 scope.go:122] "RemoveContainer" containerID="28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30" Nov 04 01:35:02.648220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.646084 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h" Nov 04 01:35:02.706037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.670891 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" event={"ID":"b7ecc416-ae43-4785-a349-eae11939a796","Type":"ContainerStarted","Data":"c9e6aeb930a6bd9b2bed44e0f5a9934703992846c60235410907f81f32acf158"} Nov 04 01:35:02.706037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.675164 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" event={"ID":"e84c9996-9944-48ca-8ca4-e0ac9d1f8c30","Type":"ContainerDied","Data":"feb798862e0ef08a4f74cfdbad4cd3f4b124439ec1f460bf03be844965fab268"} Nov 04 01:35:02.706037 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.675261 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9" Nov 04 01:35:02.878017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.761645 5051 scope.go:122] "RemoveContainer" containerID="28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30" Nov 04 01:35:02.878017 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:35:02.766706 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30\": container with ID starting with 28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30 not found: ID does not exist" containerID="28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30" Nov 04 01:35:02.878017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.766744 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30"} err="failed to get container status \"28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30\": rpc error: code = NotFound desc = could not find container \"28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30\": container with ID starting with 28590cdad32d58c5f86ce9e3fe61a091c054ecc4268e36478da2647a0324cb30 not found: ID does not exist" Nov 04 01:35:02.878017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.766768 5051 scope.go:122] "RemoveContainer" containerID="7da8a305e495bb78e08f160c2152fa1142422fa8be22b6f95719f9f17cc8e3cf" Nov 04 01:35:02.878017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.793997 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" podStartSLOduration=2.793962156 podStartE2EDuration="2.793962156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:02.792939567 +0000 UTC m=+2241.764688442" watchObservedRunningTime="2026-11-04 01:35:02.793962156 +0000 UTC m=+2241.765711028" Nov 04 01:35:02.946540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.946501 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h"] Nov 04 01:35:02.986289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:02.986240 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-82g4h"] Nov 04 01:35:03.090204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:03.053564 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9"] Nov 04 01:35:03.382687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:03.382646 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" path="/var/lib/kubelet/pods/7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d/volumes" Nov 04 01:35:03.383787 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:03.383759 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" path="/var/lib/kubelet/pods/eb08d7e7-695c-4005-8101-58b8a376eb7c/volumes" Nov 04 01:35:03.384642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:03.384622 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-586541-openshift-default-79bfdd9cb5-h7kd9"] Nov 04 01:35:03.896146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:03.886313 5051 generic.go:356] "Generic (PLEG): container finished" podID="b7ecc416-ae43-4785-a349-eae11939a796" containerID="c9e6aeb930a6bd9b2bed44e0f5a9934703992846c60235410907f81f32acf158" exitCode=0 Nov 04 01:35:03.896146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:03.886359 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" event={"ID":"b7ecc416-ae43-4785-a349-eae11939a796","Type":"ContainerDied","Data":"c9e6aeb930a6bd9b2bed44e0f5a9934703992846c60235410907f81f32acf158"} Nov 04 01:35:04.029273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.029228 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:04.029576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.029546 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.059915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.059859 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.249256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.248223 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.360150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.346298 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:04.360150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.346366 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.427187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.417220 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.504392 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.504272 5051 patch_prober.go:29] interesting pod/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:04.504757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.504643 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" podUID="fcb153fc-0c8d-4625-8b59-36344e1f3766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:04.581155 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:35:04.572833 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:35:04.786208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:04.783580 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:05.060358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:05.060297 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:05.065510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:05.065484 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:35:05.350246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:05.258368 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded" Nov 04 01:35:05.429147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:05.425142 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:05.899145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:05.894282 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" path="/var/lib/kubelet/pods/e84c9996-9944-48ca-8ca4-e0ac9d1f8c30/volumes" Nov 04 01:35:05.899145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:05.894872 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:35:05.899145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:05.894954 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:35:05.899145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:05.895009 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:35:06.113145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.110246 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:06.313355 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.297767 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:35:06.313355 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.302756 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:35:06.515583 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.515245 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:06.692690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.692598 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ecc416-ae43-4785-a349-eae11939a796-kube-api-access-nhll4\" (UniqueName: \"kubernetes.io/projected/b7ecc416-ae43-4785-a349-eae11939a796-kube-api-access-nhll4\") pod \"b7ecc416-ae43-4785-a349-eae11939a796\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " Nov 04 01:35:06.692881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.692723 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-openshift-ca\") pod \"b7ecc416-ae43-4785-a349-eae11939a796\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " Nov 04 01:35:06.692881 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.692758 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-xks-ca\") pod \"b7ecc416-ae43-4785-a349-eae11939a796\" (UID: \"b7ecc416-ae43-4785-a349-eae11939a796\") " Nov 04 01:35:06.694051 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.694022 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-openshift-ca" pod "b7ecc416-ae43-4785-a349-eae11939a796" (UID: "b7ecc416-ae43-4785-a349-eae11939a796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:06.695662 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.695627 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ecc416-ae43-4785-a349-eae11939a796-kube-api-access-nhll4" pod "b7ecc416-ae43-4785-a349-eae11939a796" (UID: "b7ecc416-ae43-4785-a349-eae11939a796"). InnerVolumeSpecName "kube-api-access-nhll4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:06.701162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.701099 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-xks-ca" pod "b7ecc416-ae43-4785-a349-eae11939a796" (UID: "b7ecc416-ae43-4785-a349-eae11939a796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:06.794520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.794487 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhll4\" (UniqueName: \"kubernetes.io/projected/b7ecc416-ae43-4785-a349-eae11939a796-kube-api-access-nhll4\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:06.794520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.794516 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:06.794520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:06.794528 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ecc416-ae43-4785-a349-eae11939a796-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:07.285195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.285111 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" event={"ID":"b7ecc416-ae43-4785-a349-eae11939a796","Type":"ContainerDied","Data":"d629579fe9fade688ea442c038e51dfcb7f9a79e8c0cbf3003b02976ab80f23d"} Nov 04 01:35:07.285195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.285202 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d629579fe9fade688ea442c038e51dfcb7f9a79e8c0cbf3003b02976ab80f23d" Nov 04 01:35:07.285717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.285259 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7" Nov 04 01:35:07.404723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.404021 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" containerID="cri-o://1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588" gracePeriod=45 Nov 04 01:35:07.406555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.405917 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" containerID="cri-o://5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21" gracePeriod=45 Nov 04 01:35:07.879690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.879657 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:35:07.882884 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.882861 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:35:07.942717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.942652 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e23a20-9d9c-4335-899b-03f9e8eee4b9-kube-api-access-kcpjc\" (UniqueName: \"kubernetes.io/projected/06e23a20-9d9c-4335-899b-03f9e8eee4b9-kube-api-access-kcpjc\") pod \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " Nov 04 01:35:07.943168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.942743 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-dshm\") pod \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " Nov 04 01:35:07.943168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.942796 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c41ff4-0264-4807-99e7-daa17e5d0f5a-kube-api-access-fgrx2\" (UniqueName: \"kubernetes.io/projected/01c41ff4-0264-4807-99e7-daa17e5d0f5a-kube-api-access-fgrx2\") pod \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " Nov 04 01:35:07.943168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.942826 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-model-cache\") pod \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " Nov 04 01:35:07.943168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.942894 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tls-certs\") pod \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " Nov 04 01:35:07.943168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.942954 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tmp-dir\") pod \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " Nov 04 01:35:07.943168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.942992 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-home\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-home\") pod \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " Nov 04 01:35:07.943168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.943029 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-dshm\") pod \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " Nov 04 01:35:07.943168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.943057 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-model-cache\") pod \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " Nov 04 01:35:07.943697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.943219 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-home\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-home\") pod \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " Nov 04 01:35:07.943697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.943283 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tls-certs\") pod \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\" (UID: \"06e23a20-9d9c-4335-899b-03f9e8eee4b9\") " Nov 04 01:35:07.943697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.943338 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tmp-dir\") pod \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\" (UID: \"01c41ff4-0264-4807-99e7-daa17e5d0f5a\") " Nov 04 01:35:07.944163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.944100 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tmp-dir" pod "01c41ff4-0264-4807-99e7-daa17e5d0f5a" (UID: "01c41ff4-0264-4807-99e7-daa17e5d0f5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:07.944737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.944710 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tmp-dir" pod "06e23a20-9d9c-4335-899b-03f9e8eee4b9" (UID: "06e23a20-9d9c-4335-899b-03f9e8eee4b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:07.946006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.945944 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-home" pod "06e23a20-9d9c-4335-899b-03f9e8eee4b9" (UID: "06e23a20-9d9c-4335-899b-03f9e8eee4b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:07.946006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.945957 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-home" pod "01c41ff4-0264-4807-99e7-daa17e5d0f5a" (UID: "01c41ff4-0264-4807-99e7-daa17e5d0f5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:07.946652 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.946616 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-model-cache" pod "06e23a20-9d9c-4335-899b-03f9e8eee4b9" (UID: "06e23a20-9d9c-4335-899b-03f9e8eee4b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:07.946927 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.946906 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-model-cache" pod "01c41ff4-0264-4807-99e7-daa17e5d0f5a" (UID: "01c41ff4-0264-4807-99e7-daa17e5d0f5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:07.968686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.968644 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-dshm" pod "06e23a20-9d9c-4335-899b-03f9e8eee4b9" (UID: "06e23a20-9d9c-4335-899b-03f9e8eee4b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:07.968686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.968649 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tls-certs" pod "01c41ff4-0264-4807-99e7-daa17e5d0f5a" (UID: "01c41ff4-0264-4807-99e7-daa17e5d0f5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:07.968903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.968782 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c41ff4-0264-4807-99e7-daa17e5d0f5a-kube-api-access-fgrx2" pod "01c41ff4-0264-4807-99e7-daa17e5d0f5a" (UID: "01c41ff4-0264-4807-99e7-daa17e5d0f5a"). InnerVolumeSpecName "kube-api-access-fgrx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:07.968949 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.968922 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e23a20-9d9c-4335-899b-03f9e8eee4b9-kube-api-access-kcpjc" pod "06e23a20-9d9c-4335-899b-03f9e8eee4b9" (UID: "06e23a20-9d9c-4335-899b-03f9e8eee4b9"). InnerVolumeSpecName "kube-api-access-kcpjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:07.969042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.969019 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-dshm" pod "01c41ff4-0264-4807-99e7-daa17e5d0f5a" (UID: "01c41ff4-0264-4807-99e7-daa17e5d0f5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:07.971846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.971811 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tls-certs" pod "06e23a20-9d9c-4335-899b-03f9e8eee4b9" (UID: "06e23a20-9d9c-4335-899b-03f9e8eee4b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:07.998297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.998249 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh"] Nov 04 01:35:07.998565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:07.998534 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" podUID="fcb153fc-0c8d-4625-8b59-36344e1f3766" containerName="istio-proxy" containerID="cri-o://7b37c817fe94bfa4ff89de1fd59c18438775864fd61fdf249bfa79c14cbcb3df" gracePeriod=30 Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045445 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045472 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045484 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045496 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045510 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpjc\" (UniqueName: \"kubernetes.io/projected/06e23a20-9d9c-4335-899b-03f9e8eee4b9-kube-api-access-kcpjc\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045520 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045529 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgrx2\" (UniqueName: \"kubernetes.io/projected/01c41ff4-0264-4807-99e7-daa17e5d0f5a-kube-api-access-fgrx2\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045538 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045547 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c41ff4-0264-4807-99e7-daa17e5d0f5a-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045555 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045564 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c41ff4-0264-4807-99e7-daa17e5d0f5a-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.045593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.045572 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e23a20-9d9c-4335-899b-03f9e8eee4b9-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:08.293740 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.293702 5051 generic.go:356] "Generic (PLEG): container finished" podID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerID="5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21" exitCode=0 Nov 04 01:35:08.294252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.293751 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" Nov 04 01:35:08.294252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.293795 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" event={"ID":"01c41ff4-0264-4807-99e7-daa17e5d0f5a","Type":"ContainerDied","Data":"5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21"} Nov 04 01:35:08.294252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.293845 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd" event={"ID":"01c41ff4-0264-4807-99e7-daa17e5d0f5a","Type":"ContainerDied","Data":"87caa55f2c796888e0d7e63f4afba4c2d555a46b1bf9113d76caee204e6d6bc8"} Nov 04 01:35:08.294252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.293864 5051 scope.go:122] "RemoveContainer" containerID="5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21" Nov 04 01:35:08.295826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.295794 5051 generic.go:356] "Generic (PLEG): container finished" podID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerID="1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588" exitCode=0 Nov 04 01:35:08.295950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.295858 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" Nov 04 01:35:08.295950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.295888 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" event={"ID":"06e23a20-9d9c-4335-899b-03f9e8eee4b9","Type":"ContainerDied","Data":"1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588"} Nov 04 01:35:08.295950 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.295916 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k" event={"ID":"06e23a20-9d9c-4335-899b-03f9e8eee4b9","Type":"ContainerDied","Data":"27dab2c839f8615e4b24f95f522a6d078206af36666560ac8c34d71a874294c2"} Nov 04 01:35:08.316439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.316416 5051 scope.go:122] "RemoveContainer" containerID="5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21" Nov 04 01:35:08.316751 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:35:08.316728 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21\": container with ID starting with 5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21 not found: ID does not exist" containerID="5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21" Nov 04 01:35:08.316847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.316756 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21"} err="failed to get container status \"5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21\": rpc error: code = NotFound desc = could not find container \"5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21\": container with ID starting with 5b2a560ab679b9afb466ce29fd97eac2bb7c66eb706fff6ab9e414210f2afb21 not found: ID does not exist" Nov 04 01:35:08.316847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.316772 5051 scope.go:122] "RemoveContainer" containerID="1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588" Nov 04 01:35:08.336529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.336510 5051 scope.go:122] "RemoveContainer" containerID="1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588" Nov 04 01:35:08.336907 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:35:08.336880 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588\": container with ID starting with 1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588 not found: ID does not exist" containerID="1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588" Nov 04 01:35:08.336997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.336913 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588"} err="failed to get container status \"1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588\": rpc error: code = NotFound desc = could not find container \"1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588\": container with ID starting with 1478c11f3125db015d1b4796084e7e8dffd6a904542f984af9aa0ce38880b588 not found: ID does not exist" Nov 04 01:35:08.337543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.337521 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k"] Nov 04 01:35:08.350641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.350607 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-w3-ca55c1-kserve-59ccb57444-gjx6k"] Nov 04 01:35:08.368189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.368165 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd"] Nov 04 01:35:08.374361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.374332 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ca55c1/distinct-2-w3-ca55c1-kserve-594cffcc7f-z64rd"] Nov 04 01:35:08.994324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994289 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t"] Nov 04 01:35:08.994818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994805 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b7ecc416-ae43-4785-a349-eae11939a796" containerName="revoke-keys" Nov 04 01:35:08.994859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994819 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ecc416-ae43-4785-a349-eae11939a796" containerName="revoke-keys" Nov 04 01:35:08.994859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994833 5051 cpu_manager.go:414] "containerMap: removing container" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" Nov 04 01:35:08.994859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994840 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" Nov 04 01:35:08.994976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994863 5051 cpu_manager.go:414] "containerMap: removing container" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" Nov 04 01:35:08.994976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994868 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" Nov 04 01:35:08.994976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994883 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" Nov 04 01:35:08.994976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994888 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" Nov 04 01:35:08.994976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994897 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" containerName="payload-processing" Nov 04 01:35:08.994976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994901 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" containerName="payload-processing" Nov 04 01:35:08.994976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994914 5051 cpu_manager.go:414] "containerMap: removing container" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" Nov 04 01:35:08.994976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994919 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" Nov 04 01:35:08.995223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994990 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ecc416-ae43-4785-a349-eae11939a796" containerName="revoke-keys" Nov 04 01:35:08.995223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.994998 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb08d7e7-695c-4005-8101-58b8a376eb7c" containerName="istio-proxy" Nov 04 01:35:08.995223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.995008 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" containerName="main" Nov 04 01:35:08.995223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.995015 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84c9996-9944-48ca-8ca4-e0ac9d1f8c30" containerName="istio-proxy" Nov 04 01:35:08.995223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.995021 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf4d5a9-5c82-4907-8a5d-4141c86a7f2d" containerName="payload-processing" Nov 04 01:35:08.995223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:08.995028 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" containerName="main" Nov 04 01:35:09.001060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.001038 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.005003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.004963 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:09.005945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.005921 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:35:09.007165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.007141 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t"] Nov 04 01:35:09.007384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.007350 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:35:09.007511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.007440 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:35:09.060761 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.060715 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh58j\" (UniqueName: \"kubernetes.io/projected/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-kube-api-access-jh58j\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.060961 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.060787 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.061055 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.060980 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.162234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.162195 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.162436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.162335 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh58j\" (UniqueName: \"kubernetes.io/projected/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-kube-api-access-jh58j\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.162436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.162372 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.162999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.162972 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.165086 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.165059 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.176492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.176465 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh58j\" (UniqueName: \"kubernetes.io/projected/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-kube-api-access-jh58j\") pod \"maas-api-revoke-sub-c99a1b31-gwb5t\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.203836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.203797 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c41ff4-0264-4807-99e7-daa17e5d0f5a" path="/var/lib/kubelet/pods/01c41ff4-0264-4807-99e7-daa17e5d0f5a/volumes" Nov 04 01:35:09.204870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.204835 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e23a20-9d9c-4335-899b-03f9e8eee4b9" path="/var/lib/kubelet/pods/06e23a20-9d9c-4335-899b-03f9e8eee4b9/volumes" Nov 04 01:35:09.324257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.324181 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:09.610782 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:09.610750 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t"] Nov 04 01:35:10.317720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.317689 5051 generic.go:356] "Generic (PLEG): container finished" podID="fcb153fc-0c8d-4625-8b59-36344e1f3766" containerID="7b37c817fe94bfa4ff89de1fd59c18438775864fd61fdf249bfa79c14cbcb3df" exitCode=0 Nov 04 01:35:10.317915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.317760 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" event={"ID":"fcb153fc-0c8d-4625-8b59-36344e1f3766","Type":"ContainerDied","Data":"7b37c817fe94bfa4ff89de1fd59c18438775864fd61fdf249bfa79c14cbcb3df"} Nov 04 01:35:10.317915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.317814 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" event={"ID":"fcb153fc-0c8d-4625-8b59-36344e1f3766","Type":"ContainerDied","Data":"64a2d3aac62a861656d45303fdd39677106bcec61f4101c8e2be238dd76acf7f"} Nov 04 01:35:10.317915 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.317831 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64a2d3aac62a861656d45303fdd39677106bcec61f4101c8e2be238dd76acf7f" Nov 04 01:35:10.319442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.319418 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" event={"ID":"0c99d7cb-b20c-4bee-ab2f-60d1923d3523","Type":"ContainerStarted","Data":"0bdec15aa292b0d01064da4664f8c6027dd27a7b765facd463ca4a861d3a5f58"} Nov 04 01:35:10.319582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.319471 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" event={"ID":"0c99d7cb-b20c-4bee-ab2f-60d1923d3523","Type":"ContainerStarted","Data":"e510fc894c22f19b6120cd6da42e75607eefbe895e4c99cd2e56ada984517403"} Nov 04 01:35:10.321987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.321969 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:35:10.339234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.339191 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" podStartSLOduration=2.339177859 podStartE2EDuration="2.339177859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:10.337390959 +0000 UTC m=+2249.309139837" watchObservedRunningTime="2026-11-04 01:35:10.339177859 +0000 UTC m=+2249.310926730" Nov 04 01:35:10.380038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.379960 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-socket\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380060 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-envoy\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380108 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-token\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-token\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380227 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-credential-socket\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380275 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-podinfo\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380336 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb153fc-0c8d-4625-8b59-36344e1f3766-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb153fc-0c8d-4625-8b59-36344e1f3766-istiod-ca-cert\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380353 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-socket" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:10.380454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380417 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-certs\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380454 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-data\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380531 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-kube-api-access-f4bq9\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-kube-api-access-f4bq9\") pod \"fcb153fc-0c8d-4625-8b59-36344e1f3766\" (UID: \"fcb153fc-0c8d-4625-8b59-36344e1f3766\") " Nov 04 01:35:10.380683 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380648 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-credential-socket" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:10.380867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.380784 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb153fc-0c8d-4625-8b59-36344e1f3766-istiod-ca-cert" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:10.381687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.381206 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-certs" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:10.381687 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.381503 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-data" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:10.382879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.382789 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.382879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.382815 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb153fc-0c8d-4625-8b59-36344e1f3766-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.382879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.382830 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.382879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.382843 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.382879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.382857 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.384991 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.384957 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-envoy" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:10.385078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.385038 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-token" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:10.385078 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.385038 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-podinfo" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:10.385347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.385323 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-kube-api-access-f4bq9" pod "fcb153fc-0c8d-4625-8b59-36344e1f3766" (UID: "fcb153fc-0c8d-4625-8b59-36344e1f3766"). InnerVolumeSpecName "kube-api-access-f4bq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:10.484192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.484092 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.484192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.484155 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.484192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.484175 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4bq9\" (UniqueName: \"kubernetes.io/projected/fcb153fc-0c8d-4625-8b59-36344e1f3766-kube-api-access-f4bq9\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:10.484192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:10.484188 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb153fc-0c8d-4625-8b59-36344e1f3766-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:11.329631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:11.329592 5051 generic.go:356] "Generic (PLEG): container finished" podID="0c99d7cb-b20c-4bee-ab2f-60d1923d3523" containerID="0bdec15aa292b0d01064da4664f8c6027dd27a7b765facd463ca4a861d3a5f58" exitCode=0 Nov 04 01:35:11.329809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:11.329673 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" event={"ID":"0c99d7cb-b20c-4bee-ab2f-60d1923d3523","Type":"ContainerDied","Data":"0bdec15aa292b0d01064da4664f8c6027dd27a7b765facd463ca4a861d3a5f58"} Nov 04 01:35:11.329809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:11.329716 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh" Nov 04 01:35:11.369549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:11.369513 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh"] Nov 04 01:35:11.376550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:11.376519 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-ef6cc9-openshift-default-9fc774869-phpbh"] Nov 04 01:35:12.643863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.643833 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:12.734655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.734622 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-xks-ca\") pod \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " Nov 04 01:35:12.734900 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.734866 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-openshift-ca\") pod \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " Nov 04 01:35:12.735064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.734925 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-kube-api-access-jh58j\" (UniqueName: \"kubernetes.io/projected/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-kube-api-access-jh58j\") pod \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\" (UID: \"0c99d7cb-b20c-4bee-ab2f-60d1923d3523\") " Nov 04 01:35:12.735444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.735400 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-openshift-ca" pod "0c99d7cb-b20c-4bee-ab2f-60d1923d3523" (UID: "0c99d7cb-b20c-4bee-ab2f-60d1923d3523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:12.737501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.737447 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-xks-ca" pod "0c99d7cb-b20c-4bee-ab2f-60d1923d3523" (UID: "0c99d7cb-b20c-4bee-ab2f-60d1923d3523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:12.739160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.738068 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-kube-api-access-jh58j" pod "0c99d7cb-b20c-4bee-ab2f-60d1923d3523" (UID: "0c99d7cb-b20c-4bee-ab2f-60d1923d3523"). InnerVolumeSpecName "kube-api-access-jh58j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:12.852100 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.852058 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:12.852381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.852360 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh58j\" (UniqueName: \"kubernetes.io/projected/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-kube-api-access-jh58j\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:12.856017 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:12.855989 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c99d7cb-b20c-4bee-ab2f-60d1923d3523-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:13.229143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:13.221799 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb153fc-0c8d-4625-8b59-36344e1f3766" path="/var/lib/kubelet/pods/fcb153fc-0c8d-4625-8b59-36344e1f3766/volumes" Nov 04 01:35:13.352442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:13.352403 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" event={"ID":"0c99d7cb-b20c-4bee-ab2f-60d1923d3523","Type":"ContainerDied","Data":"e510fc894c22f19b6120cd6da42e75607eefbe895e4c99cd2e56ada984517403"} Nov 04 01:35:13.352442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:13.352437 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t" Nov 04 01:35:13.352675 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:13.352467 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e510fc894c22f19b6120cd6da42e75607eefbe895e4c99cd2e56ada984517403" Nov 04 01:35:13.482313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:13.482274 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c"] Nov 04 01:35:14.230408 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.230373 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz"] Nov 04 01:35:14.231758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.231474 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0c99d7cb-b20c-4bee-ab2f-60d1923d3523" containerName="revoke-keys" Nov 04 01:35:14.231758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.231501 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c99d7cb-b20c-4bee-ab2f-60d1923d3523" containerName="revoke-keys" Nov 04 01:35:14.231758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.231555 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fcb153fc-0c8d-4625-8b59-36344e1f3766" containerName="istio-proxy" Nov 04 01:35:14.231758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.231565 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb153fc-0c8d-4625-8b59-36344e1f3766" containerName="istio-proxy" Nov 04 01:35:14.231758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.231714 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb153fc-0c8d-4625-8b59-36344e1f3766" containerName="istio-proxy" Nov 04 01:35:14.231758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.231743 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c99d7cb-b20c-4bee-ab2f-60d1923d3523" containerName="revoke-keys" Nov 04 01:35:14.235658 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.235638 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.269085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.269045 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz"] Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340181 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340231 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340269 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340320 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340394 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6h8\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-kube-api-access-wr6h8\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340472 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a584294-fb82-4693-9f6d-cc34422abc27-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340535 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a584294-fb82-4693-9f6d-cc34422abc27-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340588 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.341142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.340667 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.441374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441323 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.441374 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441367 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.441659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441387 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.441659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441419 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.441659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6h8\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-kube-api-access-wr6h8\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.441659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441535 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a584294-fb82-4693-9f6d-cc34422abc27-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.441659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441583 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a584294-fb82-4693-9f6d-cc34422abc27-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.441659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441628 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.442003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441684 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.442003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441763 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-credential-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.442003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441821 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-certs\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.442003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.441903 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-socket\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.442254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.442013 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-data\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.442476 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.442455 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a584294-fb82-4693-9f6d-cc34422abc27-istiod-ca-cert\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.444197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.444173 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-envoy\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.444321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.444217 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a584294-fb82-4693-9f6d-cc34422abc27-istio-podinfo\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.452101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.452049 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6h8\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-kube-api-access-wr6h8\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.452428 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.452387 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-istio-token\") pod \"e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.561758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.561718 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:14.863433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.863388 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz"] Nov 04 01:35:14.897529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.897480 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:14.897711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.897576 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:14.897711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:14.897620 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:15.394751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:15.394708 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" event={"ID":"0a584294-fb82-4693-9f6d-cc34422abc27","Type":"ContainerStarted","Data":"c7c612fe7841431ab47740b1fd146880e24a840729cfaa406178643a8275b91d"} Nov 04 01:35:15.395207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:15.394778 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" event={"ID":"0a584294-fb82-4693-9f6d-cc34422abc27","Type":"ContainerStarted","Data":"5dcfb3c51e9192bcb18a2580f2d48d575313a675659da25d3fe1ea9aa36b0e75"} Nov 04 01:35:15.416231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:15.416168 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podStartSLOduration=1.416149475 podStartE2EDuration="1.416149475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:15.415743089 +0000 UTC m=+2254.387491961" watchObservedRunningTime="2026-11-04 01:35:15.416149475 +0000 UTC m=+2254.387898346" Nov 04 01:35:15.562289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:15.562245 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:15.564072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:15.564041 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 04 01:35:15.564236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:15.564095 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 04 01:35:16.563657 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.563614 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 04 01:35:16.564307 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.564274 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 04 01:35:16.867990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.866699 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2"] Nov 04 01:35:16.883145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.882975 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:16.900794 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.900757 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6b69\" (UniqueName: \"kubernetes.io/projected/55ad96a0-3907-4264-9638-4b2ff56cd7a0-kube-api-access-w6b69\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:16.904534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.901875 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2"] Nov 04 01:35:16.904832 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.904806 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:16.905999 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.905976 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:16.906545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.906512 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:35:16.906639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.906566 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:16.906712 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.906676 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:35:16.906839 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:16.906819 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:35:17.008519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.008481 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:17.008714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.008557 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:17.008714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.008678 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6b69\" (UniqueName: \"kubernetes.io/projected/55ad96a0-3907-4264-9638-4b2ff56cd7a0-kube-api-access-w6b69\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:17.009306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.009279 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:17.011046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.011020 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:17.022466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.022436 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6b69\" (UniqueName: \"kubernetes.io/projected/55ad96a0-3907-4264-9638-4b2ff56cd7a0-kube-api-access-w6b69\") pod \"maas-api-revoke-sub-8d10a786-dmpr2\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:17.228066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.227973 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:17.557140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.549262 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2"] Nov 04 01:35:17.562789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.562739 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 04 01:35:17.563029 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:17.563001 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 04 01:35:18.418223 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:18.418188 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" event={"ID":"55ad96a0-3907-4264-9638-4b2ff56cd7a0","Type":"ContainerStarted","Data":"a01993fd80dc7bdfb69161ef1a9d3cba3672837b58dd3b69eeea742590d4c5d3"} Nov 04 01:35:18.418681 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:18.418243 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" event={"ID":"55ad96a0-3907-4264-9638-4b2ff56cd7a0","Type":"ContainerStarted","Data":"e3c35e1ead4a5fbe3e9faffb8a3e2f9e242af2668a5fb2228f33d3015b0a72aa"} Nov 04 01:35:18.468143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:18.461399 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" podStartSLOduration=2.461379356 podStartE2EDuration="2.461379356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:18.453465253 +0000 UTC m=+2257.425214127" watchObservedRunningTime="2026-11-04 01:35:18.461379356 +0000 UTC m=+2257.433128228" Nov 04 01:35:18.563622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:18.562237 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 04 01:35:18.563622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:18.562297 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 04 01:35:19.428633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:19.428595 5051 generic.go:356] "Generic (PLEG): container finished" podID="55ad96a0-3907-4264-9638-4b2ff56cd7a0" containerID="a01993fd80dc7bdfb69161ef1a9d3cba3672837b58dd3b69eeea742590d4c5d3" exitCode=0 Nov 04 01:35:19.429049 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:19.428698 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" event={"ID":"55ad96a0-3907-4264-9638-4b2ff56cd7a0","Type":"ContainerDied","Data":"a01993fd80dc7bdfb69161ef1a9d3cba3672837b58dd3b69eeea742590d4c5d3"} Nov 04 01:35:19.574656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:19.574604 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 04 01:35:19.574871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:19.574706 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 04 01:35:20.589105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.588728 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 04 01:35:20.589105 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.588779 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 04 01:35:20.728451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.728423 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:20.786351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.786299 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-xks-ca\") pod \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " Nov 04 01:35:20.786568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.786386 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ad96a0-3907-4264-9638-4b2ff56cd7a0-kube-api-access-w6b69\" (UniqueName: \"kubernetes.io/projected/55ad96a0-3907-4264-9638-4b2ff56cd7a0-kube-api-access-w6b69\") pod \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " Nov 04 01:35:20.786568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.786532 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-openshift-ca\") pod \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\" (UID: \"55ad96a0-3907-4264-9638-4b2ff56cd7a0\") " Nov 04 01:35:20.787471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.787435 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-openshift-ca" pod "55ad96a0-3907-4264-9638-4b2ff56cd7a0" (UID: "55ad96a0-3907-4264-9638-4b2ff56cd7a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:20.790488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.790450 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-xks-ca" pod "55ad96a0-3907-4264-9638-4b2ff56cd7a0" (UID: "55ad96a0-3907-4264-9638-4b2ff56cd7a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:20.792302 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.792271 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ad96a0-3907-4264-9638-4b2ff56cd7a0-kube-api-access-w6b69" pod "55ad96a0-3907-4264-9638-4b2ff56cd7a0" (UID: "55ad96a0-3907-4264-9638-4b2ff56cd7a0"). InnerVolumeSpecName "kube-api-access-w6b69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:20.888684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.888640 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:20.888684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.888677 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ad96a0-3907-4264-9638-4b2ff56cd7a0-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:20.888684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:20.888690 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6b69\" (UniqueName: \"kubernetes.io/projected/55ad96a0-3907-4264-9638-4b2ff56cd7a0-kube-api-access-w6b69\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:21.448282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:21.448191 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" event={"ID":"55ad96a0-3907-4264-9638-4b2ff56cd7a0","Type":"ContainerDied","Data":"e3c35e1ead4a5fbe3e9faffb8a3e2f9e242af2668a5fb2228f33d3015b0a72aa"} Nov 04 01:35:21.448282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:21.448234 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2" Nov 04 01:35:21.448568 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:21.448241 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3c35e1ead4a5fbe3e9faffb8a3e2f9e242af2668a5fb2228f33d3015b0a72aa" Nov 04 01:35:21.501605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:21.501568 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p"] Nov 04 01:35:21.571036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:21.570989 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Nov 04 01:35:21.571460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:21.571421 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Nov 04 01:35:22.706244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:22.706193 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:22.708731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:22.708684 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:22.711484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:22.711460 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:28.550156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:28.549425 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" containerID="cri-o://39cdc41ef49e5222314d506bf964e9266d7748547d44e2a5cc4c044f9c4a5623" gracePeriod=45 Nov 04 01:35:28.652434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:28.652380 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": EOF" Nov 04 01:35:29.428000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.427953 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": dial tcp 10.132.0.63:8000: connect: connection refused" Nov 04 01:35:29.551145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.495899 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:35:29.551145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.495945 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m"] Nov 04 01:35:29.599169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.552892 5051 cpu_manager.go:414] "containerMap: removing container" podUID="55ad96a0-3907-4264-9638-4b2ff56cd7a0" containerName="revoke-keys" Nov 04 01:35:29.599169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.552931 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ad96a0-3907-4264-9638-4b2ff56cd7a0" containerName="revoke-keys" Nov 04 01:35:29.599169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.553084 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ad96a0-3907-4264-9638-4b2ff56cd7a0" containerName="revoke-keys" Nov 04 01:35:29.599169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.557775 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m"] Nov 04 01:35:29.599169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.582215 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:29.672667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.624203 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:29.672667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.624262 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:29.672667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.624595 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbsx6\" (UniqueName: \"kubernetes.io/projected/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-kube-api-access-fbsx6\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:29.672667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.627847 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:35:29.672667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.629780 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:35:29.672667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.629876 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:35:29.672667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.629942 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:35:29.746156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.745327 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbsx6\" (UniqueName: \"kubernetes.io/projected/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-kube-api-access-fbsx6\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:29.746156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.745421 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:29.746156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.745452 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:29.971484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.939242 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:30.024203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.980737 5051 generic.go:356] "Generic (PLEG): container finished" podID="28707278-956e-4312-a887-be2241f79f79" containerID="39cdc41ef49e5222314d506bf964e9266d7748547d44e2a5cc4c044f9c4a5623" exitCode=0 Nov 04 01:35:30.024203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:29.980830 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" event={"ID":"28707278-956e-4312-a887-be2241f79f79","Type":"ContainerDied","Data":"39cdc41ef49e5222314d506bf964e9266d7748547d44e2a5cc4c044f9c4a5623"} Nov 04 01:35:30.066473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.066423 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbsx6\" (UniqueName: \"kubernetes.io/projected/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-kube-api-access-fbsx6\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:30.115166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.111845 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0bf94cb-ndh7m\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:30.399756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.399715 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:30.432397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.432368 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.644347 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28707278-956e-4312-a887-be2241f79f79-kube-api-access-2jpsp\" (UniqueName: \"kubernetes.io/projected/28707278-956e-4312-a887-be2241f79f79-kube-api-access-2jpsp\") pod \"28707278-956e-4312-a887-be2241f79f79\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.644418 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-dshm\") pod \"28707278-956e-4312-a887-be2241f79f79\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.644500 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-tmp-dir\") pod \"28707278-956e-4312-a887-be2241f79f79\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.644618 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-home\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-home\") pod \"28707278-956e-4312-a887-be2241f79f79\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.644668 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28707278-956e-4312-a887-be2241f79f79-tls-certs\" (UniqueName: \"kubernetes.io/secret/28707278-956e-4312-a887-be2241f79f79-tls-certs\") pod \"28707278-956e-4312-a887-be2241f79f79\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.644731 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-model-cache\") pod \"28707278-956e-4312-a887-be2241f79f79\" (UID: \"28707278-956e-4312-a887-be2241f79f79\") " Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.645818 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-model-cache" pod "28707278-956e-4312-a887-be2241f79f79" (UID: "28707278-956e-4312-a887-be2241f79f79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.646057 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-tmp-dir" pod "28707278-956e-4312-a887-be2241f79f79" (UID: "28707278-956e-4312-a887-be2241f79f79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.648697 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.646306 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-home" pod "28707278-956e-4312-a887-be2241f79f79" (UID: "28707278-956e-4312-a887-be2241f79f79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.649702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.648972 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28707278-956e-4312-a887-be2241f79f79-tls-certs" pod "28707278-956e-4312-a887-be2241f79f79" (UID: "28707278-956e-4312-a887-be2241f79f79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:30.654200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.652389 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28707278-956e-4312-a887-be2241f79f79-kube-api-access-2jpsp" pod "28707278-956e-4312-a887-be2241f79f79" (UID: "28707278-956e-4312-a887-be2241f79f79"). InnerVolumeSpecName "kube-api-access-2jpsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:30.669337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.669305 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-dshm" pod "28707278-956e-4312-a887-be2241f79f79" (UID: "28707278-956e-4312-a887-be2241f79f79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:30.808550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.780978 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.808550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.781023 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpsp\" (UniqueName: \"kubernetes.io/projected/28707278-956e-4312-a887-be2241f79f79-kube-api-access-2jpsp\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.808550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.781040 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.808550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.781055 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.808550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.781071 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28707278-956e-4312-a887-be2241f79f79-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.808550 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.781087 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28707278-956e-4312-a887-be2241f79f79-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:30.842818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.838206 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:35:30.842818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.842356 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:30.972584 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.969990 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx"] Nov 04 01:35:30.972778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.972722 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" containerID="cri-o://8c5d9ffb6a57b65b25891a589c2aad928800e76850445ec8af9b3a9d7f3dce3d" gracePeriod=30 Nov 04 01:35:30.984030 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.983419 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz"] Nov 04 01:35:30.984030 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.983638 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" containerID="cri-o://c7c612fe7841431ab47740b1fd146880e24a840729cfaa406178643a8275b91d" gracePeriod=30 Nov 04 01:35:30.994549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.994511 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s"] Nov 04 01:35:30.994777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:30.994748 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" containerID="cri-o://1ff057d8e1cf9711c11cc3195c497f4879a0b532e87737751897c5b0e6ef2a11" gracePeriod=30 Nov 04 01:35:31.204583 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:31.204548 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" Nov 04 01:35:31.235324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:31.231468 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" event={"ID":"28707278-956e-4312-a887-be2241f79f79","Type":"ContainerDied","Data":"5679ab25ba0d8d449471fc8e241c6952b2c727693545d3099894a719fd297957"} Nov 04 01:35:31.235324 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:31.231553 5051 scope.go:122] "RemoveContainer" containerID="39cdc41ef49e5222314d506bf964e9266d7748547d44e2a5cc4c044f9c4a5623" Nov 04 01:35:31.252314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:31.252204 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c"] Nov 04 01:35:31.257679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:31.257615 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c"] Nov 04 01:35:31.261544 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:31.261519 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m"] Nov 04 01:35:31.305881 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:35:31.305828 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b76b141_381a_42f3_9b7e_32d8e61ec5d3.slice/crio-64c6ed8712baa2760d3b9d3c5461ff6d1edf93f9def9b5136b4df8b3e6ae80f1 WatchSource:0}: Error finding container 64c6ed8712baa2760d3b9d3c5461ff6d1edf93f9def9b5136b4df8b3e6ae80f1: Status 404 returned error can't find the container with id 64c6ed8712baa2760d3b9d3c5461ff6d1edf93f9def9b5136b4df8b3e6ae80f1 Nov 04 01:35:31.311132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:31.311081 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-499ddd/simulator-w6-499ddd-kserve-6d6cc6d875-wzh6c" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 01:35:32.214353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:32.214320 5051 generic.go:356] "Generic (PLEG): container finished" podID="1b76b141-381a-42f3-9b7e-32d8e61ec5d3" containerID="542748726cc91af1593dc4b2c0da60acfaa687b9fd1d699c7d745aca1fdbf9ba" exitCode=0 Nov 04 01:35:32.214777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:32.214400 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" event={"ID":"1b76b141-381a-42f3-9b7e-32d8e61ec5d3","Type":"ContainerDied","Data":"542748726cc91af1593dc4b2c0da60acfaa687b9fd1d699c7d745aca1fdbf9ba"} Nov 04 01:35:32.214777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:32.214454 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" event={"ID":"1b76b141-381a-42f3-9b7e-32d8e61ec5d3","Type":"ContainerStarted","Data":"64c6ed8712baa2760d3b9d3c5461ff6d1edf93f9def9b5136b4df8b3e6ae80f1"} Nov 04 01:35:32.352726 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:32.352676 5051 patch_prober.go:29] interesting pod/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:35:32.352897 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:32.352749 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:35:33.189772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.189738 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28707278-956e-4312-a887-be2241f79f79" path="/var/lib/kubelet/pods/28707278-956e-4312-a887-be2241f79f79/volumes" Nov 04 01:35:33.376198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.376173 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:33.446069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.445991 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-kube-api-access-fbsx6\" (UniqueName: \"kubernetes.io/projected/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-kube-api-access-fbsx6\") pod \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " Nov 04 01:35:33.446069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.446042 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-openshift-ca\") pod \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " Nov 04 01:35:33.446277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.446235 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-xks-ca\") pod \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\" (UID: \"1b76b141-381a-42f3-9b7e-32d8e61ec5d3\") " Nov 04 01:35:33.446517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.446490 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-openshift-ca" pod "1b76b141-381a-42f3-9b7e-32d8e61ec5d3" (UID: "1b76b141-381a-42f3-9b7e-32d8e61ec5d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:33.446711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.446693 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:33.448320 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.448292 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-kube-api-access-fbsx6" pod "1b76b141-381a-42f3-9b7e-32d8e61ec5d3" (UID: "1b76b141-381a-42f3-9b7e-32d8e61ec5d3"). InnerVolumeSpecName "kube-api-access-fbsx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:33.448414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.448323 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-xks-ca" pod "1b76b141-381a-42f3-9b7e-32d8e61ec5d3" (UID: "1b76b141-381a-42f3-9b7e-32d8e61ec5d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:33.547629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.547593 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbsx6\" (UniqueName: \"kubernetes.io/projected/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-kube-api-access-fbsx6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:33.547629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:33.547625 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b76b141-381a-42f3-9b7e-32d8e61ec5d3-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:34.107004 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:35:34.106967 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:35:34.230796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:34.230759 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" Nov 04 01:35:34.230974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:34.230792 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m" event={"ID":"1b76b141-381a-42f3-9b7e-32d8e61ec5d3","Type":"ContainerDied","Data":"64c6ed8712baa2760d3b9d3c5461ff6d1edf93f9def9b5136b4df8b3e6ae80f1"} Nov 04 01:35:34.230974 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:34.230855 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64c6ed8712baa2760d3b9d3c5461ff6d1edf93f9def9b5136b4df8b3e6ae80f1" Nov 04 01:35:36.257942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.257583 5051 generic.go:356] "Generic (PLEG): container finished" podID="d4067319-b8e0-4ec9-8356-93c60b083658" containerID="8c5d9ffb6a57b65b25891a589c2aad928800e76850445ec8af9b3a9d7f3dce3d" exitCode=0 Nov 04 01:35:36.257942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.257694 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" event={"ID":"d4067319-b8e0-4ec9-8356-93c60b083658","Type":"ContainerDied","Data":"8c5d9ffb6a57b65b25891a589c2aad928800e76850445ec8af9b3a9d7f3dce3d"} Nov 04 01:35:36.257942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.257784 5051 scope.go:122] "RemoveContainer" containerID="b1350ae90d59a02fae6f1c684aa0b9988173658e9280448e9c8136f6ecfec681" Nov 04 01:35:36.261775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.261704 5051 generic.go:356] "Generic (PLEG): container finished" podID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerID="1ff057d8e1cf9711c11cc3195c497f4879a0b532e87737751897c5b0e6ef2a11" exitCode=0 Nov 04 01:35:36.261910 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.261829 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" event={"ID":"70adf13e-81e4-46a0-96ae-09ba8b3ba474","Type":"ContainerDied","Data":"1ff057d8e1cf9711c11cc3195c497f4879a0b532e87737751897c5b0e6ef2a11"} Nov 04 01:35:36.263776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.263748 5051 generic.go:356] "Generic (PLEG): container finished" podID="0a584294-fb82-4693-9f6d-cc34422abc27" containerID="c7c612fe7841431ab47740b1fd146880e24a840729cfaa406178643a8275b91d" exitCode=0 Nov 04 01:35:36.266059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.263942 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" event={"ID":"0a584294-fb82-4693-9f6d-cc34422abc27","Type":"ContainerDied","Data":"c7c612fe7841431ab47740b1fd146880e24a840729cfaa406178643a8275b91d"} Nov 04 01:35:36.350818 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.350780 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433241 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-istio-token\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-istio-token\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433308 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-certs\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433420 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-credential-socket\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433461 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4067319-b8e0-4ec9-8356-93c60b083658-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4067319-b8e0-4ec9-8356-93c60b083658-istio-podinfo\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433632 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-socket\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433665 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-kube-api-access-qfptb\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-kube-api-access-qfptb\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433712 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4067319-b8e0-4ec9-8356-93c60b083658-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4067319-b8e0-4ec9-8356-93c60b083658-istiod-ca-cert\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433742 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-data\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.434165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.433780 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-envoy\") pod \"d4067319-b8e0-4ec9-8356-93c60b083658\" (UID: \"d4067319-b8e0-4ec9-8356-93c60b083658\") " Nov 04 01:35:36.448467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.448426 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-certs" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.449899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.449689 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-socket" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.450290 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.450249 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4067319-b8e0-4ec9-8356-93c60b083658-istiod-ca-cert" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.450414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.450344 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-envoy" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.450612 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.450548 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-credential-socket" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.450768 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.450738 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-data" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.455891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.455864 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4067319-b8e0-4ec9-8356-93c60b083658-istio-podinfo" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.456308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.456285 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-istio-token" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.456632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.456592 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-kube-api-access-qfptb" pod "d4067319-b8e0-4ec9-8356-93c60b083658" (UID: "d4067319-b8e0-4ec9-8356-93c60b083658"). InnerVolumeSpecName "kube-api-access-qfptb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.463598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.463570 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:36.465784 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.465710 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:35:36.534143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.525840 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" containerID="cri-o://a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9" gracePeriod=45 Nov 04 01:35:36.543863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543831 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543864 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4067319-b8e0-4ec9-8356-93c60b083658-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543880 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543892 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfptb\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-kube-api-access-qfptb\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543907 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4067319-b8e0-4ec9-8356-93c60b083658-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543921 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543934 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543947 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4067319-b8e0-4ec9-8356-93c60b083658-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.543961 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4067319-b8e0-4ec9-8356-93c60b083658-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.645413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645316 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istiod-ca-cert\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.645413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645376 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-credential-socket\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.645413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645411 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-envoy\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.645711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645449 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-data\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.645711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645487 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-credential-socket\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.645711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645537 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-data\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.645711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645586 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-kube-api-access-wr6h8\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-kube-api-access-wr6h8\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.645711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645628 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-certs\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.645711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645696 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-podinfo\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.646052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645724 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a584294-fb82-4693-9f6d-cc34422abc27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a584294-fb82-4693-9f6d-cc34422abc27-istio-podinfo\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.646052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645773 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istiod-ca-cert" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.646052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645797 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-token\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-token\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.646052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645840 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-istio-token\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-istio-token\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.646052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645877 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-socket\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.646052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.645934 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a584294-fb82-4693-9f6d-cc34422abc27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a584294-fb82-4693-9f6d-cc34422abc27-istiod-ca-cert\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.646052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.646001 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-socket\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.646052 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.646031 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-kube-api-access-l8wx5\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-kube-api-access-l8wx5\") pod \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\" (UID: \"70adf13e-81e4-46a0-96ae-09ba8b3ba474\") " Nov 04 01:35:36.646515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.646052 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-data" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.646515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.646161 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-certs\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.646515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.646196 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-envoy\") pod \"0a584294-fb82-4693-9f6d-cc34422abc27\" (UID: \"0a584294-fb82-4693-9f6d-cc34422abc27\") " Nov 04 01:35:36.646877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.646621 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-data" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.646877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.646823 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-credential-socket" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.647178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.647010 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.647178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.647027 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.647178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.647043 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.647178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.647042 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-credential-socket" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.647178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.647060 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.647464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.647314 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-certs" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.648245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.648195 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-certs" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.649476 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.649442 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-socket" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.650196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.650169 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-kube-api-access-l8wx5" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "kube-api-access-l8wx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.650535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.650509 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a584294-fb82-4693-9f6d-cc34422abc27-istiod-ca-cert" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:36.650634 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.650579 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-envoy" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.650855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.650832 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-socket" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.651115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.651090 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-token" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.651831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.651807 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-kube-api-access-wr6h8" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "kube-api-access-wr6h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.652287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.652229 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-envoy" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:36.652456 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.652432 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a584294-fb82-4693-9f6d-cc34422abc27-istio-podinfo" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.653064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.653042 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-podinfo" pod "70adf13e-81e4-46a0-96ae-09ba8b3ba474" (UID: "70adf13e-81e4-46a0-96ae-09ba8b3ba474"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:36.653328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.653305 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-istio-token" pod "0a584294-fb82-4693-9f6d-cc34422abc27" (UID: "0a584294-fb82-4693-9f6d-cc34422abc27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770626 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770666 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770680 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770694 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770708 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr6h8\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-kube-api-access-wr6h8\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770723 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770737 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770749 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a584294-fb82-4693-9f6d-cc34422abc27-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770763 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770778 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a584294-fb82-4693-9f6d-cc34422abc27-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770791 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a584294-fb82-4693-9f6d-cc34422abc27-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770807 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a584294-fb82-4693-9f6d-cc34422abc27-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770820 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70adf13e-81e4-46a0-96ae-09ba8b3ba474-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.774173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.770833 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8wx5\" (UniqueName: \"kubernetes.io/projected/70adf13e-81e4-46a0-96ae-09ba8b3ba474-kube-api-access-l8wx5\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:36.945350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.945317 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:35:36.995217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.995179 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b70f01-982e-49b9-84d3-76efa67498e9-kube-api-access-l6gnv\" (UniqueName: \"kubernetes.io/projected/02b70f01-982e-49b9-84d3-76efa67498e9-kube-api-access-l6gnv\") pod \"02b70f01-982e-49b9-84d3-76efa67498e9\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " Nov 04 01:35:36.995506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.995479 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-home\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-home\") pod \"02b70f01-982e-49b9-84d3-76efa67498e9\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " Nov 04 01:35:36.995689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.995671 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-tmp-dir\") pod \"02b70f01-982e-49b9-84d3-76efa67498e9\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " Nov 04 01:35:36.996004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.995982 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b70f01-982e-49b9-84d3-76efa67498e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/02b70f01-982e-49b9-84d3-76efa67498e9-tls-certs\") pod \"02b70f01-982e-49b9-84d3-76efa67498e9\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " Nov 04 01:35:36.996242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.996223 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-dshm\") pod \"02b70f01-982e-49b9-84d3-76efa67498e9\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " Nov 04 01:35:36.996502 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.996482 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-model-cache\") pod \"02b70f01-982e-49b9-84d3-76efa67498e9\" (UID: \"02b70f01-982e-49b9-84d3-76efa67498e9\") " Nov 04 01:35:36.997447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:36.997402 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-model-cache" pod "02b70f01-982e-49b9-84d3-76efa67498e9" (UID: "02b70f01-982e-49b9-84d3-76efa67498e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:37.001270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.000787 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b70f01-982e-49b9-84d3-76efa67498e9-kube-api-access-l6gnv" pod "02b70f01-982e-49b9-84d3-76efa67498e9" (UID: "02b70f01-982e-49b9-84d3-76efa67498e9"). InnerVolumeSpecName "kube-api-access-l6gnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:37.001583 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.001477 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-home" pod "02b70f01-982e-49b9-84d3-76efa67498e9" (UID: "02b70f01-982e-49b9-84d3-76efa67498e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:37.002399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.002362 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-tmp-dir" pod "02b70f01-982e-49b9-84d3-76efa67498e9" (UID: "02b70f01-982e-49b9-84d3-76efa67498e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:37.008749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.008710 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-dshm" pod "02b70f01-982e-49b9-84d3-76efa67498e9" (UID: "02b70f01-982e-49b9-84d3-76efa67498e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:37.008984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.008948 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b70f01-982e-49b9-84d3-76efa67498e9-tls-certs" pod "02b70f01-982e-49b9-84d3-76efa67498e9" (UID: "02b70f01-982e-49b9-84d3-76efa67498e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:35:37.098395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.098229 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.098395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.098273 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.098395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.098290 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6gnv\" (UniqueName: \"kubernetes.io/projected/02b70f01-982e-49b9-84d3-76efa67498e9-kube-api-access-l6gnv\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.098395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.098319 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.098395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.098334 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b70f01-982e-49b9-84d3-76efa67498e9-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.098395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.098348 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02b70f01-982e-49b9-84d3-76efa67498e9-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:37.279203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.277496 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" Nov 04 01:35:37.279203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.278324 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx" event={"ID":"d4067319-b8e0-4ec9-8356-93c60b083658","Type":"ContainerDied","Data":"a4b240984306c0189ad40140445e2361c1046cf7e21b0e8b5b2f180532497437"} Nov 04 01:35:37.279203 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.278378 5051 scope.go:122] "RemoveContainer" containerID="8c5d9ffb6a57b65b25891a589c2aad928800e76850445ec8af9b3a9d7f3dce3d" Nov 04 01:35:37.284448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.284217 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" Nov 04 01:35:37.284448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.284244 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s" event={"ID":"70adf13e-81e4-46a0-96ae-09ba8b3ba474","Type":"ContainerDied","Data":"1e174c81a9a22bd684f5da5b78cee600cf25023838b14bd7fd84737873e1423f"} Nov 04 01:35:37.294196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.291619 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" event={"ID":"0a584294-fb82-4693-9f6d-cc34422abc27","Type":"ContainerDied","Data":"5dcfb3c51e9192bcb18a2580f2d48d575313a675659da25d3fe1ea9aa36b0e75"} Nov 04 01:35:37.294196 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.291775 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz" Nov 04 01:35:37.297446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.297415 5051 generic.go:356] "Generic (PLEG): container finished" podID="02b70f01-982e-49b9-84d3-76efa67498e9" containerID="a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9" exitCode=0 Nov 04 01:35:37.297599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.297537 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" event={"ID":"02b70f01-982e-49b9-84d3-76efa67498e9","Type":"ContainerDied","Data":"a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9"} Nov 04 01:35:37.297599 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.297570 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" event={"ID":"02b70f01-982e-49b9-84d3-76efa67498e9","Type":"ContainerDied","Data":"5f80715ee856a311f2995616539caf9b65d7b292ae888e433345fcc85e27a864"} Nov 04 01:35:37.297729 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.297631 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" Nov 04 01:35:37.307491 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.307427 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx"] Nov 04 01:35:37.327846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.322389 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-6sfwx"] Nov 04 01:35:37.335311 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.335174 5051 scope.go:122] "RemoveContainer" containerID="1ff057d8e1cf9711c11cc3195c497f4879a0b532e87737751897c5b0e6ef2a11" Nov 04 01:35:37.356973 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.356936 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p"] Nov 04 01:35:37.365384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.365317 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p"] Nov 04 01:35:37.403114 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.402738 5051 scope.go:122] "RemoveContainer" containerID="c7c612fe7841431ab47740b1fd146880e24a840729cfaa406178643a8275b91d" Nov 04 01:35:37.419210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.414606 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz"] Nov 04 01:35:37.426140 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.420709 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-mjhwz"] Nov 04 01:35:37.431643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.431613 5051 scope.go:122] "RemoveContainer" containerID="a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9" Nov 04 01:35:37.432439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.432301 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s"] Nov 04 01:35:37.437217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.437192 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ca55c1-openshift-default-6494847f9f-t4x5s"] Nov 04 01:35:37.448358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.448332 5051 scope.go:122] "RemoveContainer" containerID="a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9" Nov 04 01:35:37.448956 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:35:37.448903 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9\": container with ID starting with a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9 not found: ID does not exist" containerID="a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9" Nov 04 01:35:37.449282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.449171 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9"} err="failed to get container status \"a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9\": rpc error: code = NotFound desc = could not find container \"a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9\": container with ID starting with a97819fa27bcd03a499a77418e57c85d809b1d4f7fdc0c5aba9c13b3c77a31b9 not found: ID does not exist" Nov 04 01:35:37.780144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:37.770026 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d7e45d/distinct-2-w2-d7e45d-kserve-74b56fcbd5-7c29p" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded" Nov 04 01:35:39.193660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:39.193623 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" path="/var/lib/kubelet/pods/02b70f01-982e-49b9-84d3-76efa67498e9/volumes" Nov 04 01:35:39.195425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:39.195395 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" path="/var/lib/kubelet/pods/0a584294-fb82-4693-9f6d-cc34422abc27/volumes" Nov 04 01:35:39.196270 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:39.196246 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" path="/var/lib/kubelet/pods/70adf13e-81e4-46a0-96ae-09ba8b3ba474/volumes" Nov 04 01:35:39.197874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:39.197355 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" path="/var/lib/kubelet/pods/d4067319-b8e0-4ec9-8356-93c60b083658/volumes" Nov 04 01:35:44.457667 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.457618 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx"] Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458324 5051 cpu_manager.go:414] "containerMap: removing container" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458346 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458364 5051 cpu_manager.go:414] "containerMap: removing container" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458370 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458383 5051 cpu_manager.go:414] "containerMap: removing container" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458390 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458405 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458410 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458425 5051 cpu_manager.go:414] "containerMap: removing container" podUID="1b76b141-381a-42f3-9b7e-32d8e61ec5d3" containerName="revoke-keys" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458429 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b76b141-381a-42f3-9b7e-32d8e61ec5d3" containerName="revoke-keys" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458442 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458450 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458548 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="70adf13e-81e4-46a0-96ae-09ba8b3ba474" containerName="istio-proxy" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458556 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="28707278-956e-4312-a887-be2241f79f79" containerName="main" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458563 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b70f01-982e-49b9-84d3-76efa67498e9" containerName="main" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458570 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458578 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b76b141-381a-42f3-9b7e-32d8e61ec5d3" containerName="revoke-keys" Nov 04 01:35:44.458826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.458584 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a584294-fb82-4693-9f6d-cc34422abc27" containerName="istio-proxy" Nov 04 01:35:44.463263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.463236 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.485097 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.477573 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx"] Nov 04 01:35:44.519836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.519799 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.520026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.519864 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7e6635-b044-4487-a8a0-a374029f92e5-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.520026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.519966 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.520197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.520036 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.520197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.520172 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.520284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.520234 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.520371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.520353 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8g9\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-kube-api-access-xk8g9\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.520409 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.520385 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.520445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.520422 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.621448 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.621403 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8g9\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-kube-api-access-xk8g9\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.621756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.621709 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.622008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.621983 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.622251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.622229 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.622384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.622370 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7e6635-b044-4487-a8a0-a374029f92e5-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.622501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.622488 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.622638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.622625 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.622769 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.622753 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.622901 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.622885 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.623592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.623558 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.646847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.646815 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-podinfo\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.647388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.647363 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-data\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.647780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.647761 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-certs\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.648620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.648584 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7e6635-b044-4487-a8a0-a374029f92e5-istiod-ca-cert\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.649006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.648988 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-credential-socket\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.652873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.652851 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-envoy\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.660164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.653344 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8g9\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-kube-api-access-xk8g9\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.660164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.659301 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-token\") pod \"e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:44.788746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:44.788707 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:45.261252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:45.260417 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:45.261252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:45.260520 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:45.261252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:45.260569 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:35:45.274180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:45.274075 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx"] Nov 04 01:35:45.402155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:45.397630 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" event={"ID":"cf7e6635-b044-4487-a8a0-a374029f92e5","Type":"ContainerStarted","Data":"c83248328315e00622f1dac8d098c34b1608567c10abd6844aa4b5aa96cc783e"} Nov 04 01:35:46.407220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:46.407175 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" event={"ID":"cf7e6635-b044-4487-a8a0-a374029f92e5","Type":"ContainerStarted","Data":"bdd0a3e1cd33650df87cd111813839b6e5498493693cf2f8bca33fb0df38614c"} Nov 04 01:35:46.430258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:46.430203 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" podStartSLOduration=2.430189382 podStartE2EDuration="2.430189382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:35:46.426203864 +0000 UTC m=+2285.397952736" watchObservedRunningTime="2026-11-04 01:35:46.430189382 +0000 UTC m=+2285.401938253" Nov 04 01:35:46.790056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:46.790024 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:46.791837 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:46.791796 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 04 01:35:46.791995 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:46.791867 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 04 01:35:47.790083 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:47.790028 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 04 01:35:47.790532 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:47.790140 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 04 01:35:48.790056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:48.790021 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 04 01:35:48.790235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:48.790099 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 04 01:35:49.377838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:49.377790 5051 scope.go:122] "RemoveContainer" containerID="cf70477f5a627238340f036429f8106d2717fb9edcd68b4541c6fb24528d0c08" Nov 04 01:35:49.389298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:49.389279 5051 scope.go:122] "RemoveContainer" containerID="564494348f5efde3da2fcb9721d08b2e46ee51b95cda91d8ba67aa5310f033c4" Nov 04 01:35:49.399333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:49.399307 5051 scope.go:122] "RemoveContainer" containerID="7b37c817fe94bfa4ff89de1fd59c18438775864fd61fdf249bfa79c14cbcb3df" Nov 04 01:35:49.409359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:49.409337 5051 scope.go:122] "RemoveContainer" containerID="bcde942b5da34a582b28c4c96c6cca53d4e9cb3a0063bfe1b69fafee331a6fb2" Nov 04 01:35:49.790379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:49.790340 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 04 01:35:49.790804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:49.790397 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 04 01:35:50.790166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:50.790114 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 04 01:35:50.790357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:50.790191 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 04 01:35:51.790115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:51.790027 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Nov 04 01:35:51.790551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:51.790106 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Nov 04 01:35:52.793701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:52.793663 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:52.794245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:52.794039 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:52.794809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:52.794791 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:35:53.288731 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:53.288681 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt"] Nov 04 01:35:53.288972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:53.288940 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" containerID="cri-o://83438193f00a9a199f20d489f8c4e3fb7aef72898b4aef9bf302433a96d17874" gracePeriod=30 Nov 04 01:35:53.295617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:53.295582 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf"] Nov 04 01:35:53.296313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:53.296246 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" containerID="cri-o://d5d0844cd1e594173bb0e0ec07af3ef7297c79d366db28537947136a87db7935" gracePeriod=30 Nov 04 01:35:53.299859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:53.299835 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s"] Nov 04 01:35:53.300062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:53.300015 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" containerID="cri-o://e9656b4705c35f93e16133532d0ec840f29d296d2fd4732da334aa749c2e4210" gracePeriod=30 Nov 04 01:35:56.339478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:56.339436 5051 patch_prober.go:29] interesting pod/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:35:56.339893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:56.339491 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:35:58.550236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.549993 5051 generic.go:356] "Generic (PLEG): container finished" podID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerID="e9656b4705c35f93e16133532d0ec840f29d296d2fd4732da334aa749c2e4210" exitCode=0 Nov 04 01:35:58.550236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.550145 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" event={"ID":"d79c50b2-a884-4533-a240-cc18c3a65bb9","Type":"ContainerDied","Data":"e9656b4705c35f93e16133532d0ec840f29d296d2fd4732da334aa749c2e4210"} Nov 04 01:35:58.553722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.553542 5051 generic.go:356] "Generic (PLEG): container finished" podID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerID="83438193f00a9a199f20d489f8c4e3fb7aef72898b4aef9bf302433a96d17874" exitCode=0 Nov 04 01:35:58.553722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.553658 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" event={"ID":"b20b9fc8-c661-4782-9739-33c09ef7b15c","Type":"ContainerDied","Data":"83438193f00a9a199f20d489f8c4e3fb7aef72898b4aef9bf302433a96d17874"} Nov 04 01:35:58.556614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.556331 5051 generic.go:356] "Generic (PLEG): container finished" podID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerID="d5d0844cd1e594173bb0e0ec07af3ef7297c79d366db28537947136a87db7935" exitCode=0 Nov 04 01:35:58.556614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.556438 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" event={"ID":"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6","Type":"ContainerDied","Data":"d5d0844cd1e594173bb0e0ec07af3ef7297c79d366db28537947136a87db7935"} Nov 04 01:35:58.556614 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.556480 5051 scope.go:122] "RemoveContainer" containerID="96c6c0a31fb77cfe64fe167823a1e8511fe0d916b6e4f81505c345905878c5e6" Nov 04 01:35:58.642698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.642316 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:35:58.705898 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.705872 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:35:58.764060 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764018 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istiod-ca-cert\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764083 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-credential-socket\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764151 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-socket\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764190 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-certs\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764286 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-podinfo\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764373 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-data\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764414 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-envoy\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764465 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-kube-api-access-wwlv8\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-kube-api-access-wwlv8\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764513 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-token\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-token\") pod \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\" (UID: \"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6\") " Nov 04 01:35:58.764714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764563 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istiod-ca-cert" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:58.765001 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.764979 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.766090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.765376 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-socket" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.766090 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.766057 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-data" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.766463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.766427 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-credential-socket" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.768237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.768209 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-certs" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.768676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.768635 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-token" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.768766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.768738 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-podinfo" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:58.769200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.769158 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-kube-api-access-wwlv8" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "kube-api-access-wwlv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.770073 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.770045 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-envoy" pod "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" (UID: "fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.866310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866279 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-envoy\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866367 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-certs\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866401 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-socket\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866438 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-data\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866463 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-credential-socket\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866492 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-token\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-token\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866542 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-kube-api-access-7kpmk\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-kube-api-access-7kpmk\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866560 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d79c50b2-a884-4533-a240-cc18c3a65bb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c50b2-a884-4533-a240-cc18c3a65bb9-istiod-ca-cert\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866644 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866592 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-podinfo\") pod \"d79c50b2-a884-4533-a240-cc18c3a65bb9\" (UID: \"d79c50b2-a884-4533-a240-cc18c3a65bb9\") " Nov 04 01:35:58.866939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866751 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-certs" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.866939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866769 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-socket" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.866939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.866791 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-data" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.867108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867039 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-credential-socket" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.867248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867198 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867228 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867245 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867262 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867277 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867291 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867306 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867320 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867335 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867342 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79c50b2-a884-4533-a240-cc18c3a65bb9-istiod-ca-cert" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867349 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867405 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.867758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.867459 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwlv8\" (UniqueName: \"kubernetes.io/projected/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6-kube-api-access-wwlv8\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.868609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.868580 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-envoy" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:58.869050 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.869009 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-podinfo" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:58.869459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.869432 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-kube-api-access-7kpmk" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "kube-api-access-7kpmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.869556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.869441 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-token" pod "d79c50b2-a884-4533-a240-cc18c3a65bb9" (UID: "d79c50b2-a884-4533-a240-cc18c3a65bb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:58.968107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.968077 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.968107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.968106 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.968107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.968116 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kpmk\" (UniqueName: \"kubernetes.io/projected/d79c50b2-a884-4533-a240-cc18c3a65bb9-kube-api-access-7kpmk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.968358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.968141 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79c50b2-a884-4533-a240-cc18c3a65bb9-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.968358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.968151 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79c50b2-a884-4533-a240-cc18c3a65bb9-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:58.999253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:58.999224 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:35:59.068762 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.068706 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-envoy\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.068971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.068775 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-kube-api-access-wsh5c\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-kube-api-access-wsh5c\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.068971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.068804 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-certs\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.068971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.068828 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-credential-socket\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.068971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.068851 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-podinfo\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.068971 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.068894 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-token\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-token\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.069370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069006 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-data\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.069370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069184 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20b9fc8-c661-4782-9739-33c09ef7b15c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20b9fc8-c661-4782-9739-33c09ef7b15c-istiod-ca-cert\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.069370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069222 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-socket\") pod \"b20b9fc8-c661-4782-9739-33c09ef7b15c\" (UID: \"b20b9fc8-c661-4782-9739-33c09ef7b15c\") " Nov 04 01:35:59.069370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069239 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-credential-socket" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.069370 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069250 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-certs" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.069670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069423 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-data" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.069670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069544 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-socket" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.069670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069561 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20b9fc8-c661-4782-9739-33c09ef7b15c-istiod-ca-cert" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:35:59.069843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069768 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b20b9fc8-c661-4782-9739-33c09ef7b15c-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.069843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069788 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.069843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069803 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.069843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069818 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.069843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.069831 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.071244 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.071216 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-token" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:59.071365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.071242 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-envoy" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:35:59.071365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.071267 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-podinfo" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:35:59.071453 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.071387 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-kube-api-access-wsh5c" pod "b20b9fc8-c661-4782-9739-33c09ef7b15c" (UID: "b20b9fc8-c661-4782-9739-33c09ef7b15c"). InnerVolumeSpecName "kube-api-access-wsh5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:35:59.171560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.171473 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.171560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.171506 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsh5c\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-kube-api-access-wsh5c\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.171560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.171518 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.171560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.171527 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b20b9fc8-c661-4782-9739-33c09ef7b15c-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:35:59.564487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.564447 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" Nov 04 01:35:59.565084 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.565047 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s" event={"ID":"d79c50b2-a884-4533-a240-cc18c3a65bb9","Type":"ContainerDied","Data":"2d9b2450896a7db2553121345a1e5f0f958035e6734d252678b055c094beaa12"} Nov 04 01:35:59.565256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.565197 5051 scope.go:122] "RemoveContainer" containerID="e9656b4705c35f93e16133532d0ec840f29d296d2fd4732da334aa749c2e4210" Nov 04 01:35:59.566826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.566798 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" Nov 04 01:35:59.566943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.566800 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt" event={"ID":"b20b9fc8-c661-4782-9739-33c09ef7b15c","Type":"ContainerDied","Data":"cb3757e805a3087f51d7b3b200774e7801f1a3d6cb31e0cc51640ca1c775aff8"} Nov 04 01:35:59.569261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.568714 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" Nov 04 01:35:59.569261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.568809 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf" event={"ID":"fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6","Type":"ContainerDied","Data":"536573e70935305a7a758a0c8b86cec6caf1444da92da3cc5707b6358438f424"} Nov 04 01:35:59.576904 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.576884 5051 scope.go:122] "RemoveContainer" containerID="83438193f00a9a199f20d489f8c4e3fb7aef72898b4aef9bf302433a96d17874" Nov 04 01:35:59.588530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.588492 5051 scope.go:122] "RemoveContainer" containerID="d5d0844cd1e594173bb0e0ec07af3ef7297c79d366db28537947136a87db7935" Nov 04 01:35:59.590280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.590252 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s"] Nov 04 01:35:59.593610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.593590 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-8lw9s"] Nov 04 01:35:59.608115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.608093 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf"] Nov 04 01:35:59.612639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.612616 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-zpspf"] Nov 04 01:35:59.624359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.624252 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt"] Nov 04 01:35:59.630056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:35:59.630019 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-499ddd-openshift-default-68ff67fd4f-nj2pt"] Nov 04 01:36:00.460167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:00.459398 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2"] Nov 04 01:36:00.460167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:00.459663 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" containerID="cri-o://ea62e9a01b9efe612b0f85ffa8491e0147459924506595f0e9a57d0392cf092e" gracePeriod=30 Nov 04 01:36:00.467774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:00.466235 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl"] Nov 04 01:36:00.467774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:00.466483 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" containerID="cri-o://77901f77b583c6c2ee9014488fcc8d20f3f1c4f25981c5e3b6a0bcf73b0ade8c" gracePeriod=30 Nov 04 01:36:00.471094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:00.470521 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx"] Nov 04 01:36:00.471483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:00.471444 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" containerID="cri-o://bdd0a3e1cd33650df87cd111813839b6e5498493693cf2f8bca33fb0df38614c" gracePeriod=30 Nov 04 01:36:01.188494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:01.188460 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" path="/var/lib/kubelet/pods/b20b9fc8-c661-4782-9739-33c09ef7b15c/volumes" Nov 04 01:36:01.189036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:01.188992 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" path="/var/lib/kubelet/pods/d79c50b2-a884-4533-a240-cc18c3a65bb9/volumes" Nov 04 01:36:01.189463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:01.189448 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" path="/var/lib/kubelet/pods/fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6/volumes" Nov 04 01:36:01.903807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:01.903768 5051 patch_prober.go:29] interesting pod/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:36:01.903987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:01.903836 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:36:04.105816 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:36:04.105780 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:36:05.627310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:05.627171 5051 generic.go:356] "Generic (PLEG): container finished" podID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerID="bdd0a3e1cd33650df87cd111813839b6e5498493693cf2f8bca33fb0df38614c" exitCode=0 Nov 04 01:36:05.627310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:05.627271 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" event={"ID":"cf7e6635-b044-4487-a8a0-a374029f92e5","Type":"ContainerDied","Data":"bdd0a3e1cd33650df87cd111813839b6e5498493693cf2f8bca33fb0df38614c"} Nov 04 01:36:05.635148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:05.630295 5051 generic.go:356] "Generic (PLEG): container finished" podID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerID="ea62e9a01b9efe612b0f85ffa8491e0147459924506595f0e9a57d0392cf092e" exitCode=0 Nov 04 01:36:05.635148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:05.630381 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" event={"ID":"5d5de76c-942c-4e1c-bd03-367aca840dbd","Type":"ContainerDied","Data":"ea62e9a01b9efe612b0f85ffa8491e0147459924506595f0e9a57d0392cf092e"} Nov 04 01:36:05.635148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:05.630436 5051 scope.go:122] "RemoveContainer" containerID="b518baeae647a06d1a41d7f38cf8ca6dad71b7d72250c7ab94a71474e0c455c8" Nov 04 01:36:05.635148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:05.634073 5051 generic.go:356] "Generic (PLEG): container finished" podID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerID="77901f77b583c6c2ee9014488fcc8d20f3f1c4f25981c5e3b6a0bcf73b0ade8c" exitCode=0 Nov 04 01:36:05.635148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:05.634211 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" event={"ID":"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7","Type":"ContainerDied","Data":"77901f77b583c6c2ee9014488fcc8d20f3f1c4f25981c5e3b6a0bcf73b0ade8c"} Nov 04 01:36:06.112200 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.110448 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332548 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-data\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332608 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-credential-socket\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332654 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-token\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332702 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-envoy\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332769 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-certs\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332804 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-podinfo\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332853 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-socket\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332909 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istiod-ca-cert\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.333150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.332961 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-kube-api-access-7vmfh\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-kube-api-access-7vmfh\") pod \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\" (UID: \"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7\") " Nov 04 01:36:06.398207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.334404 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-socket" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.398207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.334660 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-credential-socket" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.398207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.353850 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istiod-ca-cert" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.398207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.364567 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-certs" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.398207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.374230 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-data" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.409746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.408697 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-token" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:06.447159 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.429648 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-podinfo" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.462640 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.462683 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.462698 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.462712 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.462728 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.462741 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.462754 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.473312 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:36:06.516156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.476944 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.666626 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-podinfo\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.666681 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-kube-api-access-qwc9c\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-kube-api-access-qwc9c\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.666725 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-envoy\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.666753 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-certs\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.666801 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-data\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.666833 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5de76c-942c-4e1c-bd03-367aca840dbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5de76c-942c-4e1c-bd03-367aca840dbd-istiod-ca-cert\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.666939 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7e6635-b044-4487-a8a0-a374029f92e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7e6635-b044-4487-a8a0-a374029f92e5-istiod-ca-cert\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.666989 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-socket\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.667023 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-kube-api-access-xk8g9\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-kube-api-access-xk8g9\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.667054 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-certs\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.667093 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-data\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.682312 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-certs" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.682372 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-token\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-token\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.682460 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-credential-socket\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.682506 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-socket\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.683038 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.683287 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-socket" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.685215 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.683644 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-credential-socket" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.775755 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-data" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.775867 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-data" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.788524 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-envoy\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.788782 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-token\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-token\") pod \"cf7e6635-b044-4487-a8a0-a374029f92e5\" (UID: \"cf7e6635-b044-4487-a8a0-a374029f92e5\") " Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.789079 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-podinfo\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.789343 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-credential-socket\") pod \"5d5de76c-942c-4e1c-bd03-367aca840dbd\" (UID: \"5d5de76c-942c-4e1c-bd03-367aca840dbd\") " Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.792080 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.795200 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.795230 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.795249 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.795652 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-credential-socket" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.828389 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7e6635-b044-4487-a8a0-a374029f92e5-istiod-ca-cert" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.847234 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-socket" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.850049 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-certs" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:06.881916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.850378 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5de76c-942c-4e1c-bd03-367aca840dbd-istiod-ca-cert" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:36:06.952326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.952220 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7e6635-b044-4487-a8a0-a374029f92e5-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.952326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.952273 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.952326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.952291 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.952326 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.952308 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:06.952766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:06.952330 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5de76c-942c-4e1c-bd03-367aca840dbd-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.015069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.010257 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-podinfo" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:07.015069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.010382 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-kube-api-access-xk8g9" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "kube-api-access-xk8g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:07.015069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.011440 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-envoy" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:07.015069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.013527 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-kube-api-access-7vmfh" pod "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" (UID: "5bb6dc01-056b-447b-95c6-3f02c1fa8cb7"). InnerVolumeSpecName "kube-api-access-7vmfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:07.054471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.050794 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-podinfo" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:36:07.095147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.059416 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk8g9\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-kube-api-access-xk8g9\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.120238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.120199 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vmfh\" (UniqueName: \"kubernetes.io/projected/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-kube-api-access-7vmfh\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.120486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.120465 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.120782 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.120761 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.120923 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.120908 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.121012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.103392 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-token" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:07.121103 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.103548 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-envoy" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:07.121231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.103645 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-envoy" pod "cf7e6635-b044-4487-a8a0-a374029f92e5" (UID: "cf7e6635-b044-4487-a8a0-a374029f92e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:36:07.121323 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.103802 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-kube-api-access-qwc9c" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "kube-api-access-qwc9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:07.121414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.112603 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-token" pod "5d5de76c-942c-4e1c-bd03-367aca840dbd" (UID: "5d5de76c-942c-4e1c-bd03-367aca840dbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:36:07.121505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.090713 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" Nov 04 01:36:07.137353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.088326 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" event={"ID":"cf7e6635-b044-4487-a8a0-a374029f92e5","Type":"ContainerDied","Data":"c83248328315e00622f1dac8d098c34b1608567c10abd6844aa4b5aa96cc783e"} Nov 04 01:36:07.150147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.103205 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" Nov 04 01:36:07.163155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.088436 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx" Nov 04 01:36:07.178948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.178910 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2" event={"ID":"5d5de76c-942c-4e1c-bd03-367aca840dbd","Type":"ContainerDied","Data":"11c3f1be507f4f30dfd21b4567e2c0798f8c59b080e18a8ad627aabe85ddb8b4"} Nov 04 01:36:07.186672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.179468 5051 scope.go:122] "RemoveContainer" containerID="bdd0a3e1cd33650df87cd111813839b6e5498493693cf2f8bca33fb0df38614c" Nov 04 01:36:07.241387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.238802 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwc9c\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-kube-api-access-qwc9c\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.241387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.238843 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.241387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.238860 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.241387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.238875 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7e6635-b044-4487-a8a0-a374029f92e5-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.241387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.238890 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5de76c-942c-4e1c-bd03-367aca840dbd-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:36:07.276346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.274926 5051 scope.go:122] "RemoveContainer" containerID="ea62e9a01b9efe612b0f85ffa8491e0147459924506595f0e9a57d0392cf092e" Nov 04 01:36:07.318998 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.318928 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl" event={"ID":"5bb6dc01-056b-447b-95c6-3f02c1fa8cb7","Type":"ContainerDied","Data":"ca69a642cf3e3e5bfe4fa8e1d528dd2feec780fb0a837b5fda0d16691a68793a"} Nov 04 01:36:07.364441 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.364409 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2"] Nov 04 01:36:07.435151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.416397 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-fpwz2"] Nov 04 01:36:07.435151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.417611 5051 scope.go:122] "RemoveContainer" containerID="77901f77b583c6c2ee9014488fcc8d20f3f1c4f25981c5e3b6a0bcf73b0ade8c" Nov 04 01:36:07.435556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.435462 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl"] Nov 04 01:36:07.443144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.440329 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-76rnl"] Nov 04 01:36:07.445093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.445026 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx"] Nov 04 01:36:07.459912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:07.459847 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d7e45d-openshift-default-7b545c5b59-8hwrx"] Nov 04 01:36:09.187377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:09.187343 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" path="/var/lib/kubelet/pods/5bb6dc01-056b-447b-95c6-3f02c1fa8cb7/volumes" Nov 04 01:36:09.187862 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:09.187847 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" path="/var/lib/kubelet/pods/5d5de76c-942c-4e1c-bd03-367aca840dbd/volumes" Nov 04 01:36:09.188310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:09.188297 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" path="/var/lib/kubelet/pods/cf7e6635-b044-4487-a8a0-a374029f92e5/volumes" Nov 04 01:36:13.025510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:13.025478 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7"] Nov 04 01:36:13.028252 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:13.028223 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f17a39-jtfx7"] Nov 04 01:36:13.187678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:13.187648 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ed8bbb-c795-428c-9bf0-0e0367034761" path="/var/lib/kubelet/pods/a1ed8bbb-c795-428c-9bf0-0e0367034761/volumes" Nov 04 01:36:34.105748 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:36:34.105709 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:36:38.023103 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:38.023069 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69"] Nov 04 01:36:38.026047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:38.026017 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc1541-m2c69"] Nov 04 01:36:39.188057 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:36:39.188023 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="848d3e60-4a1e-438c-b5ad-b348c2f9f4b9" path="/var/lib/kubelet/pods/848d3e60-4a1e-438c-b5ad-b348c2f9f4b9/volumes" Nov 04 01:37:04.104606 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:37:04.104560 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:37:34.237080 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:37:34.236864 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:37:43.129547 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:37:43.129521 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:37:43.151530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:37:43.151506 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:37:49.588144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:37:49.581314 5051 scope.go:122] "RemoveContainer" containerID="510443b518b51739ac0837f76bb220d765cb28201cadc142f07591625dfdd368" Nov 04 01:37:49.628860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:37:49.628834 5051 scope.go:122] "RemoveContainer" containerID="f58e5ae022688ee9638728632ed6520f2a7ac86cf6f691f51538c9402d55d4b1" Nov 04 01:37:56.028531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:37:56.028489 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7"] Nov 04 01:37:56.032359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:37:56.032331 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c0897e-z87g7"] Nov 04 01:37:57.188253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:37:57.188214 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92673cc-c7ba-471c-a50d-a9864552e991" path="/var/lib/kubelet/pods/d92673cc-c7ba-471c-a50d-a9864552e991/volumes" Nov 04 01:38:03.939146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:03.936708 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:38:03.939146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:03.936807 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:38:04.122958 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:38:04.122877 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:38:26.033145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:26.031554 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:38:26.033145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:26.031725 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:38:26.036147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:26.035519 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:38:26.036147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:26.035576 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:38:34.106239 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:38:34.106198 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:38:44.024582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:44.024543 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp"] Nov 04 01:38:44.029306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:44.029279 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c5bf2bd-4cgkp"] Nov 04 01:38:45.188757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:45.188717 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6" path="/var/lib/kubelet/pods/a1486dd4-4274-4e46-8ed1-d2d9ee2ab6a6/volumes" Nov 04 01:38:52.541607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.541528 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld"] Nov 04 01:38:52.542059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.541732 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" podUID="bc9a5635-0458-44df-8272-31bb8a86c81c" containerName="discovery" containerID="cri-o://8ff0fc184255dadb352573957259b0d77e4191f2d04ff76e62accad08f2a9ffb" gracePeriod=30 Nov 04 01:38:52.660787 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.660761 5051 generic.go:356] "Generic (PLEG): container finished" podID="bc9a5635-0458-44df-8272-31bb8a86c81c" containerID="8ff0fc184255dadb352573957259b0d77e4191f2d04ff76e62accad08f2a9ffb" exitCode=0 Nov 04 01:38:52.660899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.660807 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" event={"ID":"bc9a5635-0458-44df-8272-31bb8a86c81c","Type":"ContainerDied","Data":"8ff0fc184255dadb352573957259b0d77e4191f2d04ff76e62accad08f2a9ffb"} Nov 04 01:38:52.811562 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.811491 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:38:52.835101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.835068 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-kubeconfig\") pod \"bc9a5635-0458-44df-8272-31bb8a86c81c\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " Nov 04 01:38:52.835101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.835164 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-kube-api-access-4kttw\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-kube-api-access-4kttw\") pod \"bc9a5635-0458-44df-8272-31bb8a86c81c\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " Nov 04 01:38:52.835101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.835194 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-dns-cert\") pod \"bc9a5635-0458-44df-8272-31bb8a86c81c\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " Nov 04 01:38:52.835101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.835227 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc9a5635-0458-44df-8272-31bb8a86c81c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9a5635-0458-44df-8272-31bb8a86c81c-local-certs\") pod \"bc9a5635-0458-44df-8272-31bb8a86c81c\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " Nov 04 01:38:52.835775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.835289 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-ca-configmap\") pod \"bc9a5635-0458-44df-8272-31bb8a86c81c\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " Nov 04 01:38:52.835775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.835375 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-cacerts\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-cacerts\") pod \"bc9a5635-0458-44df-8272-31bb8a86c81c\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " Nov 04 01:38:52.835775 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.835409 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-token\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-token\") pod \"bc9a5635-0458-44df-8272-31bb8a86c81c\" (UID: \"bc9a5635-0458-44df-8272-31bb8a86c81c\") " Nov 04 01:38:52.840187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.836497 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-ca-configmap" pod "bc9a5635-0458-44df-8272-31bb8a86c81c" (UID: "bc9a5635-0458-44df-8272-31bb8a86c81c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:38:52.840187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.838486 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-token" pod "bc9a5635-0458-44df-8272-31bb8a86c81c" (UID: "bc9a5635-0458-44df-8272-31bb8a86c81c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:52.840187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.838774 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-kubeconfig" pod "bc9a5635-0458-44df-8272-31bb8a86c81c" (UID: "bc9a5635-0458-44df-8272-31bb8a86c81c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:52.840473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.840205 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-cacerts" pod "bc9a5635-0458-44df-8272-31bb8a86c81c" (UID: "bc9a5635-0458-44df-8272-31bb8a86c81c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:52.841214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.841164 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc9a5635-0458-44df-8272-31bb8a86c81c-local-certs" pod "bc9a5635-0458-44df-8272-31bb8a86c81c" (UID: "bc9a5635-0458-44df-8272-31bb8a86c81c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:38:52.843553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.843528 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-dns-cert" pod "bc9a5635-0458-44df-8272-31bb8a86c81c" (UID: "bc9a5635-0458-44df-8272-31bb8a86c81c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:38:52.845672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.845641 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-kube-api-access-4kttw" pod "bc9a5635-0458-44df-8272-31bb8a86c81c" (UID: "bc9a5635-0458-44df-8272-31bb8a86c81c"). InnerVolumeSpecName "kube-api-access-4kttw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:38:52.937107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.937074 5051 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-ca-configmap\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:52.937107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.937103 5051 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-cacerts\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:52.937313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.937113 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:52.937313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.937151 5051 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-kubeconfig\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:52.937313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.937159 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kttw\" (UniqueName: \"kubernetes.io/projected/bc9a5635-0458-44df-8272-31bb8a86c81c-kube-api-access-4kttw\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:52.937313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.937170 5051 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc9a5635-0458-44df-8272-31bb8a86c81c-istio-csr-dns-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:52.937313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:52.937178 5051 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc9a5635-0458-44df-8272-31bb8a86c81c-local-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:38:53.669539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:53.669505 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" Nov 04 01:38:53.670068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:53.669537 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld" event={"ID":"bc9a5635-0458-44df-8272-31bb8a86c81c","Type":"ContainerDied","Data":"c17768d4ed3128d5cea4085639f33deb927d76b634e08b28caa738e84c3d8411"} Nov 04 01:38:53.670068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:53.669599 5051 scope.go:122] "RemoveContainer" containerID="8ff0fc184255dadb352573957259b0d77e4191f2d04ff76e62accad08f2a9ffb" Nov 04 01:38:53.695269 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:53.695240 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld"] Nov 04 01:38:53.702989 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:53.702961 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fjkld"] Nov 04 01:38:55.190422 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:55.190377 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9a5635-0458-44df-8272-31bb8a86c81c" path="/var/lib/kubelet/pods/bc9a5635-0458-44df-8272-31bb8a86c81c/volumes" Nov 04 01:38:56.039198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:56.039166 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb"] Nov 04 01:38:56.044187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:56.044035 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-952175fa-lwtkb"] Nov 04 01:38:57.029619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:57.029533 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd"] Nov 04 01:38:57.032154 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:57.032106 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965a29d2-cj5zd"] Nov 04 01:38:57.238892 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:57.238857 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be4d3cd-d337-4e2d-b812-5c52a9007508" path="/var/lib/kubelet/pods/0be4d3cd-d337-4e2d-b812-5c52a9007508/volumes" Nov 04 01:38:57.239789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:57.239757 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bab7ad-9dad-458b-84fb-e00dcca78644" path="/var/lib/kubelet/pods/f0bab7ad-9dad-458b-84fb-e00dcca78644/volumes" Nov 04 01:38:59.857059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857025 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9"] Nov 04 01:38:59.857479 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857466 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" Nov 04 01:38:59.857520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857479 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" Nov 04 01:38:59.857520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857497 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" Nov 04 01:38:59.857520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857502 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" Nov 04 01:38:59.857520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857508 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" Nov 04 01:38:59.857520 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857514 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857522 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857527 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857533 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857538 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857553 5051 cpu_manager.go:414] "containerMap: removing container" podUID="bc9a5635-0458-44df-8272-31bb8a86c81c" containerName="discovery" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857558 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9a5635-0458-44df-8272-31bb8a86c81c" containerName="discovery" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857572 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857577 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857586 5051 cpu_manager.go:414] "containerMap: removing container" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857591 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857599 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857603 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" Nov 04 01:38:59.857684 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857688 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9a5635-0458-44df-8272-31bb8a86c81c" containerName="discovery" Nov 04 01:38:59.858104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857698 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" Nov 04 01:38:59.858104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857706 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4067319-b8e0-4ec9-8356-93c60b083658" containerName="istio-proxy" Nov 04 01:38:59.858104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857713 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20b9fc8-c661-4782-9739-33c09ef7b15c" containerName="istio-proxy" Nov 04 01:38:59.858104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857720 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" Nov 04 01:38:59.858104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857727 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79c50b2-a884-4533-a240-cc18c3a65bb9" containerName="istio-proxy" Nov 04 01:38:59.858104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857732 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb6dc01-056b-447b-95c6-3f02c1fa8cb7" containerName="istio-proxy" Nov 04 01:38:59.858104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857738 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" Nov 04 01:38:59.858104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.857744 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7e6635-b044-4487-a8a0-a374029f92e5" containerName="istio-proxy" Nov 04 01:38:59.862611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.862591 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:38:59.867548 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.867523 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9"] Nov 04 01:38:59.867979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.867953 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:38:59.868101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.868067 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:38:59.869158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.869070 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:38:59.869306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.869204 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:38:59.990721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.990685 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:38:59.990887 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.990750 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mpw2\" (UniqueName: \"kubernetes.io/projected/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-kube-api-access-2mpw2\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:38:59.990939 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:38:59.990888 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:00.092172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.092129 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:00.092383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.092214 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:00.092383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.092253 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mpw2\" (UniqueName: \"kubernetes.io/projected/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-kube-api-access-2mpw2\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:00.092860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.092836 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:00.094510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.094485 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:00.100576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.100554 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mpw2\" (UniqueName: \"kubernetes.io/projected/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-kube-api-access-2mpw2\") pod \"maas-api-revoke-sub-a4aef7d7-p76z9\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:00.183650 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.183573 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:00.462576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.462545 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9"] Nov 04 01:39:00.464751 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:39:00.464722 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbed5ea_9eef_4068_ac83_e8ee4a45e64c.slice/crio-0aec9482f24d80b2ddb04bd9c2fff6a9fd84419d6536390cde650a0ad6c0e3c9 WatchSource:0}: Error finding container 0aec9482f24d80b2ddb04bd9c2fff6a9fd84419d6536390cde650a0ad6c0e3c9: Status 404 returned error can't find the container with id 0aec9482f24d80b2ddb04bd9c2fff6a9fd84419d6536390cde650a0ad6c0e3c9 Nov 04 01:39:00.738108 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:00.738074 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" event={"ID":"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c","Type":"ContainerStarted","Data":"0aec9482f24d80b2ddb04bd9c2fff6a9fd84419d6536390cde650a0ad6c0e3c9"} Nov 04 01:39:01.748962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:01.748913 5051 generic.go:356] "Generic (PLEG): container finished" podID="cbbed5ea-9eef-4068-ac83-e8ee4a45e64c" containerID="f41727a6be62b10c9a95f6f8969d797b433dae9125cae8cacf7ec570d2dd8558" exitCode=0 Nov 04 01:39:01.749518 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:01.749142 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" event={"ID":"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c","Type":"ContainerDied","Data":"f41727a6be62b10c9a95f6f8969d797b433dae9125cae8cacf7ec570d2dd8558"} Nov 04 01:39:02.895948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:02.895918 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:03.024282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.024193 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-xks-ca\") pod \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " Nov 04 01:39:03.024282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.024251 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-kube-api-access-2mpw2\" (UniqueName: \"kubernetes.io/projected/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-kube-api-access-2mpw2\") pod \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " Nov 04 01:39:03.024481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.024366 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-openshift-ca\") pod \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\" (UID: \"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c\") " Nov 04 01:39:03.024751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.024725 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-openshift-ca" pod "cbbed5ea-9eef-4068-ac83-e8ee4a45e64c" (UID: "cbbed5ea-9eef-4068-ac83-e8ee4a45e64c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:39:03.026412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.026382 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-xks-ca" pod "cbbed5ea-9eef-4068-ac83-e8ee4a45e64c" (UID: "cbbed5ea-9eef-4068-ac83-e8ee4a45e64c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:39:03.026524 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.026481 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-kube-api-access-2mpw2" pod "cbbed5ea-9eef-4068-ac83-e8ee4a45e64c" (UID: "cbbed5ea-9eef-4068-ac83-e8ee4a45e64c"). InnerVolumeSpecName "kube-api-access-2mpw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:39:03.125822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.125787 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:03.125822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.125821 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:03.125822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.125832 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mpw2\" (UniqueName: \"kubernetes.io/projected/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c-kube-api-access-2mpw2\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:39:03.765110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.765079 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" Nov 04 01:39:03.765333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.765078 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9" event={"ID":"cbbed5ea-9eef-4068-ac83-e8ee4a45e64c","Type":"ContainerDied","Data":"0aec9482f24d80b2ddb04bd9c2fff6a9fd84419d6536390cde650a0ad6c0e3c9"} Nov 04 01:39:03.765333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:03.765173 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aec9482f24d80b2ddb04bd9c2fff6a9fd84419d6536390cde650a0ad6c0e3c9" Nov 04 01:39:04.106953 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:39:04.106851 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:39:15.037012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:15.036965 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx"] Nov 04 01:39:15.056978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:15.056949 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eed80c0-h8rvx"] Nov 04 01:39:15.250145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:15.229483 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c31cd4-164b-450a-a31b-2f50f99f3e21" path="/var/lib/kubelet/pods/30c31cd4-164b-450a-a31b-2f50f99f3e21/volumes" Nov 04 01:39:16.061144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:16.060185 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph"] Nov 04 01:39:16.070156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:16.062940 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d02f17e-vwfph"] Nov 04 01:39:17.187607 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:17.187577 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e350fada-9253-4694-98e8-6fbd8729f39f" path="/var/lib/kubelet/pods/e350fada-9253-4694-98e8-6fbd8729f39f/volumes" Nov 04 01:39:29.023284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:29.023234 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h"] Nov 04 01:39:29.026877 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:29.026843 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66826164-mrz5h"] Nov 04 01:39:29.192781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:29.192738 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc360357-11a4-4403-8d0a-2d07854092e2" path="/var/lib/kubelet/pods/bc360357-11a4-4403-8d0a-2d07854092e2/volumes" Nov 04 01:39:32.035769 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:32.035701 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn"] Nov 04 01:39:32.047646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:32.047602 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ca5db96-zqqpn"] Nov 04 01:39:33.196815 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:33.196772 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee166bef-3997-4f6a-a9be-61035433e2e0" path="/var/lib/kubelet/pods/ee166bef-3997-4f6a-a9be-61035433e2e0/volumes" Nov 04 01:39:34.108336 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:39:34.108299 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:39:49.808660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:49.808624 5051 scope.go:122] "RemoveContainer" containerID="f1e73be98e8e4931d19162b7f21b47551b95e6c5f86a4ea200af933d75c6ea43" Nov 04 01:39:49.817608 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:49.817590 5051 scope.go:122] "RemoveContainer" containerID="b07167e5cef79bf2c049474c33888ec727b719342419a961c8d31ad4abe00ffe" Nov 04 01:39:52.026361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:52.026289 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47"] Nov 04 01:39:52.026836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:52.026814 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3c34c34-8fj47"] Nov 04 01:39:53.228145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:39:53.214691 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef7c944-67e0-4582-93e1-6b2e383e56c0" path="/var/lib/kubelet/pods/eef7c944-67e0-4582-93e1-6b2e383e56c0/volumes" Nov 04 01:40:04.106027 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:40:04.105982 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:40:07.021972 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:07.021903 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7"] Nov 04 01:40:07.024272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:07.024243 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97b0e6ba-dlkx7"] Nov 04 01:40:07.187350 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:07.187316 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ecc416-ae43-4785-a349-eae11939a796" path="/var/lib/kubelet/pods/b7ecc416-ae43-4785-a349-eae11939a796/volumes" Nov 04 01:40:13.023442 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:13.023402 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t"] Nov 04 01:40:13.026444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:13.026419 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c99a1b31-gwb5t"] Nov 04 01:40:13.188091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:13.188054 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c99d7cb-b20c-4bee-ab2f-60d1923d3523" path="/var/lib/kubelet/pods/0c99d7cb-b20c-4bee-ab2f-60d1923d3523/volumes" Nov 04 01:40:21.031146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:21.022788 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2"] Nov 04 01:40:21.031146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:21.030631 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d10a786-dmpr2"] Nov 04 01:40:21.196805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:21.196768 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ad96a0-3907-4264-9638-4b2ff56cd7a0" path="/var/lib/kubelet/pods/55ad96a0-3907-4264-9638-4b2ff56cd7a0/volumes" Nov 04 01:40:34.026930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:34.026825 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m"] Nov 04 01:40:34.030560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:34.030501 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0bf94cb-ndh7m"] Nov 04 01:40:34.118766 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:40:34.118704 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:40:35.220524 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:35.220038 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b76b141-381a-42f3-9b7e-32d8e61ec5d3" path="/var/lib/kubelet/pods/1b76b141-381a-42f3-9b7e-32d8e61ec5d3/volumes" Nov 04 01:40:49.852838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:49.852800 5051 scope.go:122] "RemoveContainer" containerID="7d681f80e448e2d7692d4a7c5f861da2f2df06644b9af26180b3d6b806c14561" Nov 04 01:40:49.862460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:49.862435 5051 scope.go:122] "RemoveContainer" containerID="0a97445506d18008252b1099e8185aa35764de8da07ba03fb0b70f4b1f977077" Nov 04 01:40:49.870792 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:49.870772 5051 scope.go:122] "RemoveContainer" containerID="d58acad9433389a875457796b42e58e9308039161204fd1545380b87025f5e2f" Nov 04 01:40:49.878867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:49.878847 5051 scope.go:122] "RemoveContainer" containerID="52ba3c040c07f1a05825b49620d3658e4a3bc6bc5a6a062b70f15da07cf6a4cd" Nov 04 01:40:49.886648 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:49.886630 5051 scope.go:122] "RemoveContainer" containerID="a12780aa7e3ecdc56a9a7667dfd274158673ec900320ed45991ac1a04b347d26" Nov 04 01:40:49.894651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:49.894632 5051 scope.go:122] "RemoveContainer" containerID="f770bfead0e59e525d9fa704c1447aa8dadbe3683c2d8c9ca7a2a6578a69b0cb" Nov 04 01:40:49.902698 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:49.902682 5051 scope.go:122] "RemoveContainer" containerID="4bb97f8c68b05e7f8afdef0a3262691abe0c59d164e8d8c4200be55c325f4598" Nov 04 01:40:52.703059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:52.703026 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh"] Nov 04 01:40:52.703555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:52.703285 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" podUID="e7701537-4386-4a18-bf86-43d4d08a758e" containerName="discovery" containerID="cri-o://6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b" gracePeriod=30 Nov 04 01:40:52.971111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:52.971087 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:40:53.078054 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.078025 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-istio-token\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-istio-token\") pod \"e7701537-4386-4a18-bf86-43d4d08a758e\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " Nov 04 01:40:53.078230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.078076 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7701537-4386-4a18-bf86-43d4d08a758e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7701537-4386-4a18-bf86-43d4d08a758e-local-certs\") pod \"e7701537-4386-4a18-bf86-43d4d08a758e\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " Nov 04 01:40:53.078230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.078149 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-ca-configmap\") pod \"e7701537-4386-4a18-bf86-43d4d08a758e\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " Nov 04 01:40:53.078230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.078174 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-kubeconfig\") pod \"e7701537-4386-4a18-bf86-43d4d08a758e\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " Nov 04 01:40:53.078230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.078193 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-dns-cert\") pod \"e7701537-4386-4a18-bf86-43d4d08a758e\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " Nov 04 01:40:53.078230 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.078221 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-kube-api-access-v7486\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-kube-api-access-v7486\") pod \"e7701537-4386-4a18-bf86-43d4d08a758e\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " Nov 04 01:40:53.078515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.078259 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-cacerts\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-cacerts\") pod \"e7701537-4386-4a18-bf86-43d4d08a758e\" (UID: \"e7701537-4386-4a18-bf86-43d4d08a758e\") " Nov 04 01:40:53.078637 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.078609 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-ca-configmap" pod "e7701537-4386-4a18-bf86-43d4d08a758e" (UID: "e7701537-4386-4a18-bf86-43d4d08a758e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:40:53.080519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.080480 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7701537-4386-4a18-bf86-43d4d08a758e-local-certs" pod "e7701537-4386-4a18-bf86-43d4d08a758e" (UID: "e7701537-4386-4a18-bf86-43d4d08a758e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:40:53.080674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.080577 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-istio-token" pod "e7701537-4386-4a18-bf86-43d4d08a758e" (UID: "e7701537-4386-4a18-bf86-43d4d08a758e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:53.080799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.080779 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-kube-api-access-v7486" pod "e7701537-4386-4a18-bf86-43d4d08a758e" (UID: "e7701537-4386-4a18-bf86-43d4d08a758e"). InnerVolumeSpecName "kube-api-access-v7486". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:40:53.080893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.080860 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-kubeconfig" pod "e7701537-4386-4a18-bf86-43d4d08a758e" (UID: "e7701537-4386-4a18-bf86-43d4d08a758e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:53.081007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.080936 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-cacerts" pod "e7701537-4386-4a18-bf86-43d4d08a758e" (UID: "e7701537-4386-4a18-bf86-43d4d08a758e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:53.081007 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.080957 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-dns-cert" pod "e7701537-4386-4a18-bf86-43d4d08a758e" (UID: "e7701537-4386-4a18-bf86-43d4d08a758e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:40:53.179075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.179043 5051 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-ca-configmap\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:53.179075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.179071 5051 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-kubeconfig\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:53.179075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.179082 5051 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-istio-csr-dns-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:53.179334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.179091 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7486\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-kube-api-access-v7486\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:53.179334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.179101 5051 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7701537-4386-4a18-bf86-43d4d08a758e-cacerts\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:53.179334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.179110 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7701537-4386-4a18-bf86-43d4d08a758e-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:53.179334 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.179146 5051 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7701537-4386-4a18-bf86-43d4d08a758e-local-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:40:53.826878 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.826840 5051 generic.go:356] "Generic (PLEG): container finished" podID="e7701537-4386-4a18-bf86-43d4d08a758e" containerID="6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b" exitCode=0 Nov 04 01:40:53.827348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.826901 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" Nov 04 01:40:53.827348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.826923 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" event={"ID":"e7701537-4386-4a18-bf86-43d4d08a758e","Type":"ContainerDied","Data":"6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b"} Nov 04 01:40:53.827348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.826979 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh" event={"ID":"e7701537-4386-4a18-bf86-43d4d08a758e","Type":"ContainerDied","Data":"4974b88e100aea3c3a6f7d5ad33499633cb905d18f5aeae6f7c3c0b28003a883"} Nov 04 01:40:53.827348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.827007 5051 scope.go:122] "RemoveContainer" containerID="6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b" Nov 04 01:40:53.839099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.839074 5051 scope.go:122] "RemoveContainer" containerID="6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b" Nov 04 01:40:53.839391 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:40:53.839371 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b\": container with ID starting with 6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b not found: ID does not exist" containerID="6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b" Nov 04 01:40:53.839478 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.839400 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b"} err="failed to get container status \"6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b\": rpc error: code = NotFound desc = could not find container \"6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b\": container with ID starting with 6841e3546a9bee30578936758539b2246cc265e977b1ad1eae301376b013e80b not found: ID does not exist" Nov 04 01:40:53.845530 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.845508 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh"] Nov 04 01:40:53.849613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:53.849592 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mtcxh"] Nov 04 01:40:55.187433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:40:55.187401 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7701537-4386-4a18-bf86-43d4d08a758e" path="/var/lib/kubelet/pods/e7701537-4386-4a18-bf86-43d4d08a758e/volumes" Nov 04 01:41:03.536286 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.536251 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs"] Nov 04 01:41:03.536956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.536932 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e7701537-4386-4a18-bf86-43d4d08a758e" containerName="discovery" Nov 04 01:41:03.537003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.536961 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7701537-4386-4a18-bf86-43d4d08a758e" containerName="discovery" Nov 04 01:41:03.537003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.536998 5051 cpu_manager.go:414] "containerMap: removing container" podUID="cbbed5ea-9eef-4068-ac83-e8ee4a45e64c" containerName="revoke-keys" Nov 04 01:41:03.537075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.537007 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbbed5ea-9eef-4068-ac83-e8ee4a45e64c" containerName="revoke-keys" Nov 04 01:41:03.537075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.537027 5051 cpu_manager.go:414] "containerMap: removing container" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" Nov 04 01:41:03.537075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.537035 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb9b76a8-cfca-4c94-b8f1-25d0ca7b35d6" containerName="istio-proxy" Nov 04 01:41:03.537204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.537191 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5de76c-942c-4e1c-bd03-367aca840dbd" containerName="istio-proxy" Nov 04 01:41:03.537241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.537206 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7701537-4386-4a18-bf86-43d4d08a758e" containerName="discovery" Nov 04 01:41:03.537241 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.537221 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbbed5ea-9eef-4068-ac83-e8ee4a45e64c" containerName="revoke-keys" Nov 04 01:41:03.542395 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.542372 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.546649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.546625 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b9b238-openshift-default-dockercfg-lzdkt\"" Nov 04 01:41:03.551594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.550811 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs"] Nov 04 01:41:03.646795 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.646746 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-token\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.647020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.646818 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-certs\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.647020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.646874 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-podinfo\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.647020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.646932 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-credential-socket\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.647020 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.647010 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istiod-ca-cert\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.647340 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.647060 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-envoy\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.647340 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.647104 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-data\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.647340 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.647160 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-socket\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.647340 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.647190 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnpj\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-kube-api-access-8cnpj\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.748545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748514 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-certs\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.748747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748572 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-podinfo\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.748747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748626 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-credential-socket\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.748747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748672 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istiod-ca-cert\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.748747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748717 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-envoy\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748760 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-data\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748797 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-socket\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748903 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnpj\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-kube-api-access-8cnpj\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749009 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.748984 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-token\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.749442 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-credential-socket\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.749463 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-certs\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.749541 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-socket\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.749706 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-data\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.749850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.749831 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istiod-ca-cert\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.751207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.751184 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-envoy\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.751594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.751576 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-podinfo\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.756993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.756970 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-token\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.757484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.757464 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnpj\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-kube-api-access-8cnpj\") pod \"e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:03.859413 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:03.859330 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:04.104789 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:41:04.104746 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:41:04.146910 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:04.146880 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs"] Nov 04 01:41:04.149879 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:41:04.149851 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e763af_df4c_41fe_8fc5_d8677f2b4f40.slice/crio-e11c2b88707059ea89bf2822a24f450322ddd55ccb9edf9ea27ce1fa692a1713 WatchSource:0}: Error finding container e11c2b88707059ea89bf2822a24f450322ddd55ccb9edf9ea27ce1fa692a1713: Status 404 returned error can't find the container with id e11c2b88707059ea89bf2822a24f450322ddd55ccb9edf9ea27ce1fa692a1713 Nov 04 01:41:04.152472 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:04.152438 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:41:04.157113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:04.152907 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:41:04.157113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:04.153043 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:41:04.927855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:04.927813 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" event={"ID":"a4e763af-df4c-41fe-8fc5-d8677f2b4f40","Type":"ContainerStarted","Data":"367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8"} Nov 04 01:41:04.928366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:04.927883 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" event={"ID":"a4e763af-df4c-41fe-8fc5-d8677f2b4f40","Type":"ContainerStarted","Data":"e11c2b88707059ea89bf2822a24f450322ddd55ccb9edf9ea27ce1fa692a1713"} Nov 04 01:41:04.951609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:04.951560 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" podStartSLOduration=1.951547301 podStartE2EDuration="1.951547301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:04.94805958 +0000 UTC m=+2603.919808463" watchObservedRunningTime="2026-11-04 01:41:04.951547301 +0000 UTC m=+2603.923296240" Nov 04 01:41:05.860066 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:05.860034 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:05.865283 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:05.865254 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:05.933466 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:05.933437 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:05.934426 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:05.934405 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:08.517774 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.517736 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx"] Nov 04 01:41:08.521622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.521598 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:08.528444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.528420 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-93d46b73-dockercfg-rjd8w\"" Nov 04 01:41:08.528763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.528744 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b9b238\"" Nov 04 01:41:08.534702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.534679 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx"] Nov 04 01:41:08.534796 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.534729 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8"] Nov 04 01:41:08.538507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.538487 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:08.549231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.549208 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8"] Nov 04 01:41:08.614872 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.614825 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24109896-eb44-4213-88df-9eb0585dab97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx\" (UID: \"24109896-eb44-4213-88df-9eb0585dab97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:08.615224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.614948 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7w75\" (UniqueName: \"kubernetes.io/projected/24109896-eb44-4213-88df-9eb0585dab97-kube-api-access-t7w75\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx\" (UID: \"24109896-eb44-4213-88df-9eb0585dab97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:08.716328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.716290 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7w75\" (UniqueName: \"kubernetes.io/projected/24109896-eb44-4213-88df-9eb0585dab97-kube-api-access-t7w75\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx\" (UID: \"24109896-eb44-4213-88df-9eb0585dab97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:08.716511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.716333 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsd9v\" (UniqueName: \"kubernetes.io/projected/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-kube-api-access-wsd9v\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8\" (UID: \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:08.716511 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.716383 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8\" (UID: \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:08.716623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.716520 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24109896-eb44-4213-88df-9eb0585dab97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx\" (UID: \"24109896-eb44-4213-88df-9eb0585dab97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:08.717315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.717288 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24109896-eb44-4213-88df-9eb0585dab97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx\" (UID: \"24109896-eb44-4213-88df-9eb0585dab97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:08.725592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.725549 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7w75\" (UniqueName: \"kubernetes.io/projected/24109896-eb44-4213-88df-9eb0585dab97-kube-api-access-t7w75\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx\" (UID: \"24109896-eb44-4213-88df-9eb0585dab97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:08.817691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.817617 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsd9v\" (UniqueName: \"kubernetes.io/projected/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-kube-api-access-wsd9v\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8\" (UID: \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:08.817691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.817672 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8\" (UID: \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:08.818403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.818382 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8\" (UID: \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:08.826318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.826287 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsd9v\" (UniqueName: \"kubernetes.io/projected/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-kube-api-access-wsd9v\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8\" (UID: \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:08.838515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.838491 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:08.857595 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:08.857569 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:09.125727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.125703 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx"] Nov 04 01:41:09.127845 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:41:09.127815 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24109896_eb44_4213_88df_9eb0585dab97.slice/crio-b3be4b23b7e04aa9ea18742eba8f3605303adaec0900c6b903feb301dd57ae87 WatchSource:0}: Error finding container b3be4b23b7e04aa9ea18742eba8f3605303adaec0900c6b903feb301dd57ae87: Status 404 returned error can't find the container with id b3be4b23b7e04aa9ea18742eba8f3605303adaec0900c6b903feb301dd57ae87 Nov 04 01:41:09.156175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.156115 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8"] Nov 04 01:41:09.159316 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:41:09.159288 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe95e61_9df9_4c4d_b661_f7ae65b62ec5.slice/crio-1c81e5a70a7f1c17fe2208f05ff45c6f43aa8c4a0f5cf4668d3f47b97428851a WatchSource:0}: Error finding container 1c81e5a70a7f1c17fe2208f05ff45c6f43aa8c4a0f5cf4668d3f47b97428851a: Status 404 returned error can't find the container with id 1c81e5a70a7f1c17fe2208f05ff45c6f43aa8c4a0f5cf4668d3f47b97428851a Nov 04 01:41:09.959642 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.959594 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" event={"ID":"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5","Type":"ContainerStarted","Data":"3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a"} Nov 04 01:41:09.960176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.959655 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" event={"ID":"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5","Type":"ContainerStarted","Data":"1c81e5a70a7f1c17fe2208f05ff45c6f43aa8c4a0f5cf4668d3f47b97428851a"} Nov 04 01:41:09.960176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.959797 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:09.961275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.961248 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" event={"ID":"24109896-eb44-4213-88df-9eb0585dab97","Type":"ContainerStarted","Data":"f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9"} Nov 04 01:41:09.961363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.961299 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" event={"ID":"24109896-eb44-4213-88df-9eb0585dab97","Type":"ContainerStarted","Data":"b3be4b23b7e04aa9ea18742eba8f3605303adaec0900c6b903feb301dd57ae87"} Nov 04 01:41:09.961363 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.961330 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:09.978944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.978897 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" podStartSLOduration=1.978884709 podStartE2EDuration="1.978884709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:09.976205497 +0000 UTC m=+2608.947954369" watchObservedRunningTime="2026-11-04 01:41:09.978884709 +0000 UTC m=+2608.950633581" Nov 04 01:41:09.991636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:09.991593 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" podStartSLOduration=1.991579936 podStartE2EDuration="1.991579936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:09.990985858 +0000 UTC m=+2608.962734738" watchObservedRunningTime="2026-11-04 01:41:09.991579936 +0000 UTC m=+2608.963328862" Nov 04 01:41:20.935611 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:20.935577 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 04 01:41:20.936104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:20.935658 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 04 01:41:20.972048 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:20.972019 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:41:20.972231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:20.972085 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:41:21.046316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:21.046283 5051 generic.go:356] "Generic (PLEG): container finished" podID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerID="367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8" exitCode=0 Nov 04 01:41:21.046485 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:21.046323 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" event={"ID":"a4e763af-df4c-41fe-8fc5-d8677f2b4f40","Type":"ContainerDied","Data":"367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8"} Nov 04 01:41:21.046716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:21.046701 5051 scope.go:122] "RemoveContainer" containerID="367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8" Nov 04 01:41:21.048134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:21.048087 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:41:21.048226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:21.048192 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:41:21.048274 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:21.048229 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:41:22.080145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:22.065687 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" event={"ID":"a4e763af-df4c-41fe-8fc5-d8677f2b4f40","Type":"ContainerStarted","Data":"09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e"} Nov 04 01:41:22.869146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:22.865477 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:22.869146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:22.865600 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 04 01:41:22.869146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:22.865678 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 04 01:41:23.868156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:23.867504 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 04 01:41:23.868156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:23.867565 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 04 01:41:24.869145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:24.868644 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 04 01:41:24.869145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:24.868725 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 04 01:41:25.862453 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:25.862416 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 04 01:41:25.862745 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:25.862718 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 04 01:41:26.865913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:26.865866 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 04 01:41:26.866501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:26.865947 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 04 01:41:27.864194 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:27.864156 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:27.864541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:27.864516 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:27.865318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:27.865296 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:41:34.105413 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:41:34.105374 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:41:49.951725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:49.951695 5051 scope.go:122] "RemoveContainer" containerID="c9e6aeb930a6bd9b2bed44e0f5a9934703992846c60235410907f81f32acf158" Nov 04 01:41:49.960695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:49.960674 5051 scope.go:122] "RemoveContainer" containerID="a01993fd80dc7bdfb69161ef1a9d3cba3672837b58dd3b69eeea742590d4c5d3" Nov 04 01:41:49.969289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:49.969272 5051 scope.go:122] "RemoveContainer" containerID="0bdec15aa292b0d01064da4664f8c6027dd27a7b765facd463ca4a861d3a5f58" Nov 04 01:41:49.977807 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:49.977787 5051 scope.go:122] "RemoveContainer" containerID="542748726cc91af1593dc4b2c0da60acfaa687b9fd1d699c7d745aca1fdbf9ba" Nov 04 01:41:53.840083 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:53.840000 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf"] Nov 04 01:41:53.845746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:53.845724 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:53.851736 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:53.851706 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:41:53.851903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:53.851829 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:41:53.851903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:53.851862 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:41:53.852033 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:53.851916 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:41:53.855358 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:53.855334 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf"] Nov 04 01:41:54.013964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.013929 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.014176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.014044 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.014176 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.014091 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2xx\" (UniqueName: \"kubernetes.io/projected/e79eabe5-6e35-4c26-af5b-095d4c6012cd-kube-api-access-gr2xx\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.114825 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.114745 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.114825 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.114791 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2xx\" (UniqueName: \"kubernetes.io/projected/e79eabe5-6e35-4c26-af5b-095d4c6012cd-kube-api-access-gr2xx\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.115064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.114860 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.115517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.115497 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.117213 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.117190 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.123046 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.123022 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2xx\" (UniqueName: \"kubernetes.io/projected/e79eabe5-6e35-4c26-af5b-095d4c6012cd-kube-api-access-gr2xx\") pod \"maas-api-revoke-sub-845fbef0-wtqgf\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.162948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.162927 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:54.436313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:54.436263 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf"] Nov 04 01:41:54.438865 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:41:54.438831 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79eabe5_6e35_4c26_af5b_095d4c6012cd.slice/crio-8291772ba7ac8fde297aa382d3385a0782318ea45656179eec284600aad58684 WatchSource:0}: Error finding container 8291772ba7ac8fde297aa382d3385a0782318ea45656179eec284600aad58684: Status 404 returned error can't find the container with id 8291772ba7ac8fde297aa382d3385a0782318ea45656179eec284600aad58684 Nov 04 01:41:55.274405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:55.274360 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" event={"ID":"e79eabe5-6e35-4c26-af5b-095d4c6012cd","Type":"ContainerStarted","Data":"66ca42e751c4728a61416fb9dd65148f3efa8c8036df8be4d7c856a473c23768"} Nov 04 01:41:55.274816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:55.274439 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" event={"ID":"e79eabe5-6e35-4c26-af5b-095d4c6012cd","Type":"ContainerStarted","Data":"8291772ba7ac8fde297aa382d3385a0782318ea45656179eec284600aad58684"} Nov 04 01:41:55.296132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:55.296064 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" podStartSLOduration=2.296047986 podStartE2EDuration="2.296047986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:41:55.293521574 +0000 UTC m=+2654.265270446" watchObservedRunningTime="2026-11-04 01:41:55.296047986 +0000 UTC m=+2654.267796860" Nov 04 01:41:56.280930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:56.280898 5051 generic.go:356] "Generic (PLEG): container finished" podID="e79eabe5-6e35-4c26-af5b-095d4c6012cd" containerID="66ca42e751c4728a61416fb9dd65148f3efa8c8036df8be4d7c856a473c23768" exitCode=0 Nov 04 01:41:56.281339 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:56.280988 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" event={"ID":"e79eabe5-6e35-4c26-af5b-095d4c6012cd","Type":"ContainerDied","Data":"66ca42e751c4728a61416fb9dd65148f3efa8c8036df8be4d7c856a473c23768"} Nov 04 01:41:57.441287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.441256 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:57.550217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.550185 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-xks-ca\") pod \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " Nov 04 01:41:57.550398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.550282 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-openshift-ca\") pod \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " Nov 04 01:41:57.550398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.550329 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79eabe5-6e35-4c26-af5b-095d4c6012cd-kube-api-access-gr2xx\" (UniqueName: \"kubernetes.io/projected/e79eabe5-6e35-4c26-af5b-095d4c6012cd-kube-api-access-gr2xx\") pod \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\" (UID: \"e79eabe5-6e35-4c26-af5b-095d4c6012cd\") " Nov 04 01:41:57.550651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.550623 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-openshift-ca" pod "e79eabe5-6e35-4c26-af5b-095d4c6012cd" (UID: "e79eabe5-6e35-4c26-af5b-095d4c6012cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:41:57.552286 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.552262 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-xks-ca" pod "e79eabe5-6e35-4c26-af5b-095d4c6012cd" (UID: "e79eabe5-6e35-4c26-af5b-095d4c6012cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:41:57.552483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.552458 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79eabe5-6e35-4c26-af5b-095d4c6012cd-kube-api-access-gr2xx" pod "e79eabe5-6e35-4c26-af5b-095d4c6012cd" (UID: "e79eabe5-6e35-4c26-af5b-095d4c6012cd"). InnerVolumeSpecName "kube-api-access-gr2xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:41:57.651490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.651462 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:57.651490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.651490 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e79eabe5-6e35-4c26-af5b-095d4c6012cd-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:57.651640 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:57.651501 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr2xx\" (UniqueName: \"kubernetes.io/projected/e79eabe5-6e35-4c26-af5b-095d4c6012cd-kube-api-access-gr2xx\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:41:58.293362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:58.293333 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" Nov 04 01:41:58.293572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:58.293375 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf" event={"ID":"e79eabe5-6e35-4c26-af5b-095d4c6012cd","Type":"ContainerDied","Data":"8291772ba7ac8fde297aa382d3385a0782318ea45656179eec284600aad58684"} Nov 04 01:41:58.293572 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:41:58.293423 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8291772ba7ac8fde297aa382d3385a0782318ea45656179eec284600aad58684" Nov 04 01:42:04.105114 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:42:04.105069 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:42:10.716129 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:10.716083 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8"] Nov 04 01:42:10.739062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:10.739024 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx"] Nov 04 01:42:15.717134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:15.717055 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" podUID="ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" containerName="payload-processing" containerID="cri-o://3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a" gracePeriod=25 Nov 04 01:42:15.739701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:15.739661 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" podUID="24109896-eb44-4213-88df-9eb0585dab97" containerName="payload-pre-processing" containerID="cri-o://f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9" gracePeriod=25 Nov 04 01:42:20.970501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:20.970458 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" podUID="24109896-eb44-4213-88df-9eb0585dab97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:20.970943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:20.970475 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" podUID="ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:30.970766 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:30.970723 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" podUID="24109896-eb44-4213-88df-9eb0585dab97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:30.971290 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:30.970757 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" podUID="ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 01:42:34.106108 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:42:34.106066 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:42:35.995734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:35.995703 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:42:36.025099 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.025068 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:42:36.144220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.144180 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24109896-eb44-4213-88df-9eb0585dab97-kube-api-access-t7w75\" (UniqueName: \"kubernetes.io/projected/24109896-eb44-4213-88df-9eb0585dab97-kube-api-access-t7w75\") pod \"24109896-eb44-4213-88df-9eb0585dab97\" (UID: \"24109896-eb44-4213-88df-9eb0585dab97\") " Nov 04 01:42:36.144417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.144241 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-plugins-config-volume\") pod \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\" (UID: \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\") " Nov 04 01:42:36.144417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.144297 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24109896-eb44-4213-88df-9eb0585dab97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24109896-eb44-4213-88df-9eb0585dab97-plugins-config-volume\") pod \"24109896-eb44-4213-88df-9eb0585dab97\" (UID: \"24109896-eb44-4213-88df-9eb0585dab97\") " Nov 04 01:42:36.144560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.144488 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-kube-api-access-wsd9v\" (UniqueName: \"kubernetes.io/projected/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-kube-api-access-wsd9v\") pod \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\" (UID: \"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5\") " Nov 04 01:42:36.144672 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.144641 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24109896-eb44-4213-88df-9eb0585dab97-plugins-config-volume" pod "24109896-eb44-4213-88df-9eb0585dab97" (UID: "24109896-eb44-4213-88df-9eb0585dab97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:42:36.144799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.144683 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-plugins-config-volume" pod "ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" (UID: "ebe95e61-9df9-4c4d-b661-f7ae65b62ec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:42:36.144846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.144837 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:36.144893 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.144852 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24109896-eb44-4213-88df-9eb0585dab97-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:36.146417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.146397 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24109896-eb44-4213-88df-9eb0585dab97-kube-api-access-t7w75" pod "24109896-eb44-4213-88df-9eb0585dab97" (UID: "24109896-eb44-4213-88df-9eb0585dab97"). InnerVolumeSpecName "kube-api-access-t7w75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:42:36.146528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.146507 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-kube-api-access-wsd9v" pod "ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" (UID: "ebe95e61-9df9-4c4d-b661-f7ae65b62ec5"). InnerVolumeSpecName "kube-api-access-wsd9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:42:36.246024 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.245988 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsd9v\" (UniqueName: \"kubernetes.io/projected/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5-kube-api-access-wsd9v\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:36.246024 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.246022 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7w75\" (UniqueName: \"kubernetes.io/projected/24109896-eb44-4213-88df-9eb0585dab97-kube-api-access-t7w75\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:36.513930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.513840 5051 generic.go:356] "Generic (PLEG): container finished" podID="ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" containerID="3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a" exitCode=0 Nov 04 01:42:36.513930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.513886 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" Nov 04 01:42:36.514216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.513932 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" event={"ID":"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5","Type":"ContainerDied","Data":"3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a"} Nov 04 01:42:36.514216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.513997 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8" event={"ID":"ebe95e61-9df9-4c4d-b661-f7ae65b62ec5","Type":"ContainerDied","Data":"1c81e5a70a7f1c17fe2208f05ff45c6f43aa8c4a0f5cf4668d3f47b97428851a"} Nov 04 01:42:36.514216 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.514027 5051 scope.go:122] "RemoveContainer" containerID="3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a" Nov 04 01:42:36.515303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.515280 5051 generic.go:356] "Generic (PLEG): container finished" podID="24109896-eb44-4213-88df-9eb0585dab97" containerID="f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9" exitCode=0 Nov 04 01:42:36.515419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.515308 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" event={"ID":"24109896-eb44-4213-88df-9eb0585dab97","Type":"ContainerDied","Data":"f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9"} Nov 04 01:42:36.515419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.515332 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" Nov 04 01:42:36.515419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.515356 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx" event={"ID":"24109896-eb44-4213-88df-9eb0585dab97","Type":"ContainerDied","Data":"b3be4b23b7e04aa9ea18742eba8f3605303adaec0900c6b903feb301dd57ae87"} Nov 04 01:42:36.524330 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.524309 5051 scope.go:122] "RemoveContainer" containerID="3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a" Nov 04 01:42:36.524641 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:42:36.524618 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a\": container with ID starting with 3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a not found: ID does not exist" containerID="3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a" Nov 04 01:42:36.524747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.524648 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a"} err="failed to get container status \"3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a\": rpc error: code = NotFound desc = could not find container \"3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a\": container with ID starting with 3311a151ccd9357de079367a23ce886c9703d346f9bbbfbfbbdec58cfa047d2a not found: ID does not exist" Nov 04 01:42:36.524747 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.524665 5051 scope.go:122] "RemoveContainer" containerID="f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9" Nov 04 01:42:36.533833 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.533811 5051 scope.go:122] "RemoveContainer" containerID="f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9" Nov 04 01:42:36.534095 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:42:36.534070 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9\": container with ID starting with f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9 not found: ID does not exist" containerID="f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9" Nov 04 01:42:36.534224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.534099 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9"} err="failed to get container status \"f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9\": rpc error: code = NotFound desc = could not find container \"f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9\": container with ID starting with f66c64dd6f91697f5d0b613361a3bf8b6b75d6033094703efb5aa78869bf92e9 not found: ID does not exist" Nov 04 01:42:36.541810 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.541783 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8"] Nov 04 01:42:36.547558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.547528 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cd7flh8"] Nov 04 01:42:36.557671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.557641 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx"] Nov 04 01:42:36.560816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:36.560791 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575c5v2kx"] Nov 04 01:42:37.187198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:37.187167 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24109896-eb44-4213-88df-9eb0585dab97" path="/var/lib/kubelet/pods/24109896-eb44-4213-88df-9eb0585dab97/volumes" Nov 04 01:42:37.187587 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:37.187565 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" path="/var/lib/kubelet/pods/ebe95e61-9df9-4c4d-b661-f7ae65b62ec5/volumes" Nov 04 01:42:43.162383 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.162354 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:42:43.187593 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.187566 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:42:43.521689 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.521650 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc"] Nov 04 01:42:43.522184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522157 5051 cpu_manager.go:414] "containerMap: removing container" podUID="e79eabe5-6e35-4c26-af5b-095d4c6012cd" containerName="revoke-keys" Nov 04 01:42:43.522184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522174 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79eabe5-6e35-4c26-af5b-095d4c6012cd" containerName="revoke-keys" Nov 04 01:42:43.522184 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522184 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" containerName="payload-processing" Nov 04 01:42:43.522467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522192 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" containerName="payload-processing" Nov 04 01:42:43.522467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522206 5051 cpu_manager.go:414] "containerMap: removing container" podUID="24109896-eb44-4213-88df-9eb0585dab97" containerName="payload-pre-processing" Nov 04 01:42:43.522467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522212 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24109896-eb44-4213-88df-9eb0585dab97" containerName="payload-pre-processing" Nov 04 01:42:43.522467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522300 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79eabe5-6e35-4c26-af5b-095d4c6012cd" containerName="revoke-keys" Nov 04 01:42:43.522467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522312 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe95e61-9df9-4c4d-b661-f7ae65b62ec5" containerName="payload-processing" Nov 04 01:42:43.522467 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.522321 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="24109896-eb44-4213-88df-9eb0585dab97" containerName="payload-pre-processing" Nov 04 01:42:43.525671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.525652 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:43.541003 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.540979 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc"] Nov 04 01:42:43.542668 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.542642 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-93d46b73-dockercfg-khxvw\"" Nov 04 01:42:43.547179 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.547152 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b9b238\"" Nov 04 01:42:43.571558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.571520 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz"] Nov 04 01:42:43.575366 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.575347 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:43.591064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.591036 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz"] Nov 04 01:42:43.620652 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.620617 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz\" (UID: \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:43.620843 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.620692 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c850f8-6a92-4cec-98b1-a82f9d2785d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc\" (UID: \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:43.620922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.620833 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhgld\" (UniqueName: \"kubernetes.io/projected/92c850f8-6a92-4cec-98b1-a82f9d2785d2-kube-api-access-dhgld\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc\" (UID: \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:43.620961 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.620950 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btvcj\" (UniqueName: \"kubernetes.io/projected/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-kube-api-access-btvcj\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz\" (UID: \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:43.721743 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.721712 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c850f8-6a92-4cec-98b1-a82f9d2785d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc\" (UID: \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:43.721916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.721754 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhgld\" (UniqueName: \"kubernetes.io/projected/92c850f8-6a92-4cec-98b1-a82f9d2785d2-kube-api-access-dhgld\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc\" (UID: \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:43.721916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.721822 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btvcj\" (UniqueName: \"kubernetes.io/projected/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-kube-api-access-btvcj\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz\" (UID: \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:43.721916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.721857 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz\" (UID: \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:43.722522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.722496 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c850f8-6a92-4cec-98b1-a82f9d2785d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc\" (UID: \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:43.722656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.722636 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz\" (UID: \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:43.730272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.730243 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btvcj\" (UniqueName: \"kubernetes.io/projected/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-kube-api-access-btvcj\") pod \"payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz\" (UID: \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:43.730412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.730245 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhgld\" (UniqueName: \"kubernetes.io/projected/92c850f8-6a92-4cec-98b1-a82f9d2785d2-kube-api-access-dhgld\") pod \"payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc\" (UID: \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:43.838337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.838240 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:43.890150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:43.890085 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:44.120074 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.120044 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc"] Nov 04 01:42:44.121823 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:42:44.121799 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c850f8_6a92_4cec_98b1_a82f9d2785d2.slice/crio-a43a859a4cce021f5563ff5c1ecfb9c5ceb20d786681f0ae68f221c1a02191c1 WatchSource:0}: Error finding container a43a859a4cce021f5563ff5c1ecfb9c5ceb20d786681f0ae68f221c1a02191c1: Status 404 returned error can't find the container with id a43a859a4cce021f5563ff5c1ecfb9c5ceb20d786681f0ae68f221c1a02191c1 Nov 04 01:42:44.206211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.206181 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz"] Nov 04 01:42:44.207591 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:42:44.207565 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5a9c47_3877_4b04_8024_6dfe0a1cdb66.slice/crio-b10734803b245f03c994779cd8223baa2838c4118e701f9a59d750527837e10c WatchSource:0}: Error finding container b10734803b245f03c994779cd8223baa2838c4118e701f9a59d750527837e10c: Status 404 returned error can't find the container with id b10734803b245f03c994779cd8223baa2838c4118e701f9a59d750527837e10c Nov 04 01:42:44.565484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.565447 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" event={"ID":"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66","Type":"ContainerStarted","Data":"ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378"} Nov 04 01:42:44.565695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.565501 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" event={"ID":"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66","Type":"ContainerStarted","Data":"b10734803b245f03c994779cd8223baa2838c4118e701f9a59d750527837e10c"} Nov 04 01:42:44.565695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.565544 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:44.567180 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.567156 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" event={"ID":"92c850f8-6a92-4cec-98b1-a82f9d2785d2","Type":"ContainerStarted","Data":"e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f"} Nov 04 01:42:44.567300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.567192 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" event={"ID":"92c850f8-6a92-4cec-98b1-a82f9d2785d2","Type":"ContainerStarted","Data":"a43a859a4cce021f5563ff5c1ecfb9c5ceb20d786681f0ae68f221c1a02191c1"} Nov 04 01:42:44.567300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.567290 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:44.586097 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.586046 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" podStartSLOduration=1.586034784 podStartE2EDuration="1.586034784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:44.585006816 +0000 UTC m=+2703.556755688" watchObservedRunningTime="2026-11-04 01:42:44.586034784 +0000 UTC m=+2703.557783654" Nov 04 01:42:44.604745 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:44.604693 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" podStartSLOduration=1.60467808 podStartE2EDuration="1.60467808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:42:44.602073197 +0000 UTC m=+2703.573822069" watchObservedRunningTime="2026-11-04 01:42:44.60467808 +0000 UTC m=+2703.576426951" Nov 04 01:42:52.801948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.801907 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf"] Nov 04 01:42:52.807094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.807073 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:52.813483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.813108 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:42:52.813483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.813341 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:42:52.813483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.813413 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:42:52.813483 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.813457 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:42:52.814776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.814750 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf"] Nov 04 01:42:52.944220 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.944183 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:52.944417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.944227 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qmbr\" (UniqueName: \"kubernetes.io/projected/dfd80d36-06c5-4194-a827-8cb41c01f47a-kube-api-access-2qmbr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:52.944417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:52.944256 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:53.045416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.045379 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:53.045609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.045528 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:53.045609 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.045558 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qmbr\" (UniqueName: \"kubernetes.io/projected/dfd80d36-06c5-4194-a827-8cb41c01f47a-kube-api-access-2qmbr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:53.046287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.046260 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:53.048044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.048024 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:53.057649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.057590 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qmbr\" (UniqueName: \"kubernetes.io/projected/dfd80d36-06c5-4194-a827-8cb41c01f47a-kube-api-access-2qmbr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:53.123735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.123694 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:53.415665 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.415517 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf"] Nov 04 01:42:53.423228 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:42:53.423192 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd80d36_06c5_4194_a827_8cb41c01f47a.slice/crio-eb4ef542442558bb18c14716dcd01e93b259c4b9059f9d1986befa0b65da2ebb WatchSource:0}: Error finding container eb4ef542442558bb18c14716dcd01e93b259c4b9059f9d1986befa0b65da2ebb: Status 404 returned error can't find the container with id eb4ef542442558bb18c14716dcd01e93b259c4b9059f9d1986befa0b65da2ebb Nov 04 01:42:53.622116 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:53.622081 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" event={"ID":"dfd80d36-06c5-4194-a827-8cb41c01f47a","Type":"ContainerStarted","Data":"eb4ef542442558bb18c14716dcd01e93b259c4b9059f9d1986befa0b65da2ebb"} Nov 04 01:42:54.628656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:54.628559 5051 generic.go:356] "Generic (PLEG): container finished" podID="dfd80d36-06c5-4194-a827-8cb41c01f47a" containerID="1e9f3fad88dfab3692230a72874cfe7ef88586b5497fcef2df7c8ca4a98d2657" exitCode=0 Nov 04 01:42:54.629056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:54.628652 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" event={"ID":"dfd80d36-06c5-4194-a827-8cb41c01f47a","Type":"ContainerDied","Data":"1e9f3fad88dfab3692230a72874cfe7ef88586b5497fcef2df7c8ca4a98d2657"} Nov 04 01:42:55.575289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.575261 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:42:55.575462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.575327 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:42:55.781727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.781702 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:42:55.875008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.874919 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-openshift-ca\") pod \"dfd80d36-06c5-4194-a827-8cb41c01f47a\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " Nov 04 01:42:55.875008 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.874990 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd80d36-06c5-4194-a827-8cb41c01f47a-kube-api-access-2qmbr\" (UniqueName: \"kubernetes.io/projected/dfd80d36-06c5-4194-a827-8cb41c01f47a-kube-api-access-2qmbr\") pod \"dfd80d36-06c5-4194-a827-8cb41c01f47a\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " Nov 04 01:42:55.875255 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.875110 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-xks-ca\") pod \"dfd80d36-06c5-4194-a827-8cb41c01f47a\" (UID: \"dfd80d36-06c5-4194-a827-8cb41c01f47a\") " Nov 04 01:42:55.875397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.875367 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-openshift-ca" pod "dfd80d36-06c5-4194-a827-8cb41c01f47a" (UID: "dfd80d36-06c5-4194-a827-8cb41c01f47a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:42:55.877401 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.877368 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-xks-ca" pod "dfd80d36-06c5-4194-a827-8cb41c01f47a" (UID: "dfd80d36-06c5-4194-a827-8cb41c01f47a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:42:55.877503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.877431 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd80d36-06c5-4194-a827-8cb41c01f47a-kube-api-access-2qmbr" pod "dfd80d36-06c5-4194-a827-8cb41c01f47a" (UID: "dfd80d36-06c5-4194-a827-8cb41c01f47a"). InnerVolumeSpecName "kube-api-access-2qmbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:42:55.976429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.976388 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:55.976429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.976423 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qmbr\" (UniqueName: \"kubernetes.io/projected/dfd80d36-06c5-4194-a827-8cb41c01f47a-kube-api-access-2qmbr\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:55.976429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:55.976434 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfd80d36-06c5-4194-a827-8cb41c01f47a-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:42:56.641342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:56.641306 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" event={"ID":"dfd80d36-06c5-4194-a827-8cb41c01f47a","Type":"ContainerDied","Data":"eb4ef542442558bb18c14716dcd01e93b259c4b9059f9d1986befa0b65da2ebb"} Nov 04 01:42:56.641531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:56.641361 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb4ef542442558bb18c14716dcd01e93b259c4b9059f9d1986befa0b65da2ebb" Nov 04 01:42:56.641531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:42:56.641320 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf" Nov 04 01:43:02.927010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:02.926974 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc"] Nov 04 01:43:02.940096 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:02.940061 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz"] Nov 04 01:43:03.411011 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:03.410974 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf"] Nov 04 01:43:03.413062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:03.413030 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9b238-s7zmf"] Nov 04 01:43:03.777690 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:03.777640 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs"] Nov 04 01:43:03.777937 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:03.777894 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" containerID="cri-o://09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e" gracePeriod=30 Nov 04 01:43:04.107222 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:43:04.107137 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:43:05.191040 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:05.190991 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd80d36-06c5-4194-a827-8cb41c01f47a" path="/var/lib/kubelet/pods/dfd80d36-06c5-4194-a827-8cb41c01f47a/volumes" Nov 04 01:43:07.927362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:07.927288 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" podUID="92c850f8-6a92-4cec-98b1-a82f9d2785d2" containerName="payload-processing" containerID="cri-o://e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f" gracePeriod=25 Nov 04 01:43:07.941171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:07.941110 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" podUID="ed5a9c47-3877-4b04-8024-6dfe0a1cdb66" containerName="payload-pre-processing" containerID="cri-o://ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378" gracePeriod=25 Nov 04 01:43:08.216155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.215342 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:43:08.267628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.267564 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:43:08.314130 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.314075 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c850f8-6a92-4cec-98b1-a82f9d2785d2-kube-api-access-dhgld\" (UniqueName: \"kubernetes.io/projected/92c850f8-6a92-4cec-98b1-a82f9d2785d2-kube-api-access-dhgld\") pod \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\" (UID: \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\") " Nov 04 01:43:08.314343 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.314208 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c850f8-6a92-4cec-98b1-a82f9d2785d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c850f8-6a92-4cec-98b1-a82f9d2785d2-plugins-config-volume\") pod \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\" (UID: \"92c850f8-6a92-4cec-98b1-a82f9d2785d2\") " Nov 04 01:43:08.314870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.314582 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c850f8-6a92-4cec-98b1-a82f9d2785d2-plugins-config-volume" pod "92c850f8-6a92-4cec-98b1-a82f9d2785d2" (UID: "92c850f8-6a92-4cec-98b1-a82f9d2785d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:08.316605 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.316582 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c850f8-6a92-4cec-98b1-a82f9d2785d2-kube-api-access-dhgld" pod "92c850f8-6a92-4cec-98b1-a82f9d2785d2" (UID: "92c850f8-6a92-4cec-98b1-a82f9d2785d2"). InnerVolumeSpecName "kube-api-access-dhgld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:08.415876 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.415835 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-plugins-config-volume\") pod \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\" (UID: \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\") " Nov 04 01:43:08.416100 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.415891 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-kube-api-access-btvcj\" (UniqueName: \"kubernetes.io/projected/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-kube-api-access-btvcj\") pod \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\" (UID: \"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66\") " Nov 04 01:43:08.416285 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.416266 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhgld\" (UniqueName: \"kubernetes.io/projected/92c850f8-6a92-4cec-98b1-a82f9d2785d2-kube-api-access-dhgld\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:08.416347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.416289 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92c850f8-6a92-4cec-98b1-a82f9d2785d2-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:08.416347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.416311 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-plugins-config-volume" pod "ed5a9c47-3877-4b04-8024-6dfe0a1cdb66" (UID: "ed5a9c47-3877-4b04-8024-6dfe0a1cdb66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:08.418080 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.418059 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-kube-api-access-btvcj" pod "ed5a9c47-3877-4b04-8024-6dfe0a1cdb66" (UID: "ed5a9c47-3877-4b04-8024-6dfe0a1cdb66"). InnerVolumeSpecName "kube-api-access-btvcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:08.517779 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.517741 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:08.517779 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.517773 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btvcj\" (UniqueName: \"kubernetes.io/projected/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66-kube-api-access-btvcj\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:08.709911 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.709878 5051 generic.go:356] "Generic (PLEG): container finished" podID="92c850f8-6a92-4cec-98b1-a82f9d2785d2" containerID="e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f" exitCode=0 Nov 04 01:43:08.710149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.709947 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" Nov 04 01:43:08.710149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.709968 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" event={"ID":"92c850f8-6a92-4cec-98b1-a82f9d2785d2","Type":"ContainerDied","Data":"e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f"} Nov 04 01:43:08.710149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.710028 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc" event={"ID":"92c850f8-6a92-4cec-98b1-a82f9d2785d2","Type":"ContainerDied","Data":"a43a859a4cce021f5563ff5c1ecfb9c5ceb20d786681f0ae68f221c1a02191c1"} Nov 04 01:43:08.710149 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.710054 5051 scope.go:122] "RemoveContainer" containerID="e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f" Nov 04 01:43:08.711934 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.711907 5051 generic.go:356] "Generic (PLEG): container finished" podID="ed5a9c47-3877-4b04-8024-6dfe0a1cdb66" containerID="ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378" exitCode=0 Nov 04 01:43:08.712059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.711946 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" event={"ID":"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66","Type":"ContainerDied","Data":"ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378"} Nov 04 01:43:08.712059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.711956 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" Nov 04 01:43:08.712059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.711991 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz" event={"ID":"ed5a9c47-3877-4b04-8024-6dfe0a1cdb66","Type":"ContainerDied","Data":"b10734803b245f03c994779cd8223baa2838c4118e701f9a59d750527837e10c"} Nov 04 01:43:08.723472 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.723404 5051 scope.go:122] "RemoveContainer" containerID="e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f" Nov 04 01:43:08.723850 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:43:08.723821 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f\": container with ID starting with e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f not found: ID does not exist" containerID="e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f" Nov 04 01:43:08.723996 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.723860 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f"} err="failed to get container status \"e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f\": rpc error: code = NotFound desc = could not find container \"e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f\": container with ID starting with e8e3cd2ce8d8d3f7e2a435c9a8108e706add533a02bfede5fff2401d8c674a0f not found: ID does not exist" Nov 04 01:43:08.723996 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.723883 5051 scope.go:122] "RemoveContainer" containerID="ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378" Nov 04 01:43:08.738296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.736265 5051 scope.go:122] "RemoveContainer" containerID="ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378" Nov 04 01:43:08.738296 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:43:08.736888 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378\": container with ID starting with ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378 not found: ID does not exist" containerID="ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378" Nov 04 01:43:08.738296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.736924 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378"} err="failed to get container status \"ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378\": rpc error: code = NotFound desc = could not find container \"ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378\": container with ID starting with ac6904d562a2b48dea2b87d4ab30df5ec327c68d64ae22deff59ad2191f33378 not found: ID does not exist" Nov 04 01:43:08.738296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.737734 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc"] Nov 04 01:43:08.741279 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.741252 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9b238-5475549cdvq7sc"] Nov 04 01:43:08.750085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.750055 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz"] Nov 04 01:43:08.754224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:08.754196 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9b238-5575cpmxcz"] Nov 04 01:43:09.042646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.042617 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:43:09.124492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124400 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-certs\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.124492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124473 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-token\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-token\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.124744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124495 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-socket\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.124744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124536 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-envoy\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.124744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124566 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-kube-api-access-8cnpj\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-kube-api-access-8cnpj\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.124744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124624 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istiod-ca-cert\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.124744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124681 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-podinfo\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.124744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124721 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-certs" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:43:09.124744 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124731 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-credential-socket\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.125156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124765 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-data\") pod \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\" (UID: \"a4e763af-df4c-41fe-8fc5-d8677f2b4f40\") " Nov 04 01:43:09.125156 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.124860 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-socket" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:43:09.125287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.125153 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.125287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.125163 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-data" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:43:09.125287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.125177 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.125287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.125167 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istiod-ca-cert" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:43:09.125287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.125198 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-credential-socket" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:43:09.126860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.126831 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-envoy" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:43:09.126979 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.126945 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-token" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:09.127103 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.127086 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-kube-api-access-8cnpj" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "kube-api-access-8cnpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:09.127237 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.127100 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-podinfo" pod "a4e763af-df4c-41fe-8fc5-d8677f2b4f40" (UID: "a4e763af-df4c-41fe-8fc5-d8677f2b4f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:43:09.186975 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.186938 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c850f8-6a92-4cec-98b1-a82f9d2785d2" path="/var/lib/kubelet/pods/92c850f8-6a92-4cec-98b1-a82f9d2785d2/volumes" Nov 04 01:43:09.187391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.187375 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5a9c47-3877-4b04-8024-6dfe0a1cdb66" path="/var/lib/kubelet/pods/ed5a9c47-3877-4b04-8024-6dfe0a1cdb66/volumes" Nov 04 01:43:09.226738 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.226702 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.226738 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.226733 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.226738 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.226745 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cnpj\" (UniqueName: \"kubernetes.io/projected/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-kube-api-access-8cnpj\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.226980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.226757 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.226980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.226766 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.226980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.226775 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.226980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.226783 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4e763af-df4c-41fe-8fc5-d8677f2b4f40-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:09.719004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.718968 5051 generic.go:356] "Generic (PLEG): container finished" podID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerID="09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e" exitCode=0 Nov 04 01:43:09.719240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.719058 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" event={"ID":"a4e763af-df4c-41fe-8fc5-d8677f2b4f40","Type":"ContainerDied","Data":"09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e"} Nov 04 01:43:09.719240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.719108 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" Nov 04 01:43:09.719240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.719151 5051 scope.go:122] "RemoveContainer" containerID="09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e" Nov 04 01:43:09.719414 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.719137 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs" event={"ID":"a4e763af-df4c-41fe-8fc5-d8677f2b4f40","Type":"ContainerDied","Data":"e11c2b88707059ea89bf2822a24f450322ddd55ccb9edf9ea27ce1fa692a1713"} Nov 04 01:43:09.728882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.728866 5051 scope.go:122] "RemoveContainer" containerID="367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8" Nov 04 01:43:09.738217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.738104 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs"] Nov 04 01:43:09.738282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.738232 5051 scope.go:122] "RemoveContainer" containerID="09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e" Nov 04 01:43:09.738498 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:43:09.738480 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e\": container with ID starting with 09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e not found: ID does not exist" containerID="09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e" Nov 04 01:43:09.738549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.738507 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e"} err="failed to get container status \"09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e\": rpc error: code = NotFound desc = could not find container \"09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e\": container with ID starting with 09d191b920a13f466c43a79b356bd9678efb116429793e1c46976b1075993a2e not found: ID does not exist" Nov 04 01:43:09.738549 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.738525 5051 scope.go:122] "RemoveContainer" containerID="367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8" Nov 04 01:43:09.738733 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:43:09.738718 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8\": container with ID starting with 367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8 not found: ID does not exist" containerID="367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8" Nov 04 01:43:09.738779 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.738738 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8"} err="failed to get container status \"367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8\": rpc error: code = NotFound desc = could not find container \"367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8\": container with ID starting with 367f8fcc6f4ed36ab2faf4edbce903d17098772d3e1012642812a812e70034d8 not found: ID does not exist" Nov 04 01:43:09.742087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:09.742063 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b9b238-openshift-default-5ff7f7bbc75x7cs"] Nov 04 01:43:11.187804 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:11.187769 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" path="/var/lib/kubelet/pods/a4e763af-df4c-41fe-8fc5-d8677f2b4f40/volumes" Nov 04 01:43:18.969361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:18.969316 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:43:18.970142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:18.969977 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:43:20.628780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:20.628730 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w"] Nov 04 01:43:20.629325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:20.629038 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" containerID="cri-o://3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3" gracePeriod=10 Nov 04 01:43:21.095349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.095323 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:43:21.106359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.106328 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-kube-api-access-8t4rj\" (UniqueName: \"kubernetes.io/projected/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-kube-api-access-8t4rj\") pod \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\" (UID: \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\") " Nov 04 01:43:21.106517 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.106458 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-extensions-socket-volume\") pod \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\" (UID: \"5c49724a-9d0f-42ec-86c8-cf8f7229f38a\") " Nov 04 01:43:21.107303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.107279 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-extensions-socket-volume" pod "5c49724a-9d0f-42ec-86c8-cf8f7229f38a" (UID: "5c49724a-9d0f-42ec-86c8-cf8f7229f38a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:43:21.108973 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.108947 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-kube-api-access-8t4rj" pod "5c49724a-9d0f-42ec-86c8-cf8f7229f38a" (UID: "5c49724a-9d0f-42ec-86c8-cf8f7229f38a"). InnerVolumeSpecName "kube-api-access-8t4rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:43:21.208195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.208164 5051 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-extensions-socket-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:21.208195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.208193 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t4rj\" (UniqueName: \"kubernetes.io/projected/5c49724a-9d0f-42ec-86c8-cf8f7229f38a-kube-api-access-8t4rj\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:43:21.819929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.819846 5051 generic.go:356] "Generic (PLEG): container finished" podID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerID="3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3" exitCode=0 Nov 04 01:43:21.819929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.819902 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" event={"ID":"5c49724a-9d0f-42ec-86c8-cf8f7229f38a","Type":"ContainerDied","Data":"3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3"} Nov 04 01:43:21.819929 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.819908 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" Nov 04 01:43:21.820565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.819949 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w" event={"ID":"5c49724a-9d0f-42ec-86c8-cf8f7229f38a","Type":"ContainerDied","Data":"284b2b317afcdbc62b18872bf850edc4a5d7c730a1bb360e25d450e10cfed495"} Nov 04 01:43:21.820565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.819976 5051 scope.go:122] "RemoveContainer" containerID="3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3" Nov 04 01:43:21.831257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.831238 5051 scope.go:122] "RemoveContainer" containerID="d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3" Nov 04 01:43:21.840540 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.840513 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w"] Nov 04 01:43:21.842573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.842549 5051 scope.go:122] "RemoveContainer" containerID="3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3" Nov 04 01:43:21.842913 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:43:21.842893 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3\": container with ID starting with 3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3 not found: ID does not exist" containerID="3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3" Nov 04 01:43:21.842994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.842921 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3"} err="failed to get container status \"3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3\": rpc error: code = NotFound desc = could not find container \"3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3\": container with ID starting with 3c88c4956592ee6ac8513c91cc2f71605b83598704be77c38e8320bdec13ecd3 not found: ID does not exist" Nov 04 01:43:21.842994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.842940 5051 scope.go:122] "RemoveContainer" containerID="d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3" Nov 04 01:43:21.843270 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:43:21.843249 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3\": container with ID starting with d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3 not found: ID does not exist" containerID="d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3" Nov 04 01:43:21.843353 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.843278 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3"} err="failed to get container status \"d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3\": rpc error: code = NotFound desc = could not find container \"d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3\": container with ID starting with d6b4bb583aff9eb6079b3f2f96b306030d304b6308c238596fe82a91fe6ce7b3 not found: ID does not exist" Nov 04 01:43:21.844619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:21.844598 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gmc9w"] Nov 04 01:43:23.187602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:43:23.187564 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" path="/var/lib/kubelet/pods/5c49724a-9d0f-42ec-86c8-cf8f7229f38a/volumes" Nov 04 01:43:34.105247 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:43:34.105210 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:44:03.023776 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:03.023740 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9"] Nov 04 01:44:03.026701 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:03.026673 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-p76z9"] Nov 04 01:44:03.186627 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:03.186599 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbbed5ea-9eef-4068-ac83-e8ee4a45e64c" path="/var/lib/kubelet/pods/cbbed5ea-9eef-4068-ac83-e8ee4a45e64c/volumes" Nov 04 01:44:04.105885 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:44:04.105846 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:44:21.620345 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620314 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb"] Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620724 5051 cpu_manager.go:414] "containerMap: removing container" podUID="ed5a9c47-3877-4b04-8024-6dfe0a1cdb66" containerName="payload-pre-processing" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620736 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5a9c47-3877-4b04-8024-6dfe0a1cdb66" containerName="payload-pre-processing" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620751 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620756 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620763 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620768 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620780 5051 cpu_manager.go:414] "containerMap: removing container" podUID="92c850f8-6a92-4cec-98b1-a82f9d2785d2" containerName="payload-processing" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620785 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c850f8-6a92-4cec-98b1-a82f9d2785d2" containerName="payload-processing" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620814 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620819 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620825 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dfd80d36-06c5-4194-a827-8cb41c01f47a" containerName="revoke-keys" Nov 04 01:44:21.620822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620829 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd80d36-06c5-4194-a827-8cb41c01f47a" containerName="revoke-keys" Nov 04 01:44:21.621282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620899 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" Nov 04 01:44:21.621282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620905 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd80d36-06c5-4194-a827-8cb41c01f47a" containerName="revoke-keys" Nov 04 01:44:21.621282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620913 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" Nov 04 01:44:21.621282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620919 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5a9c47-3877-4b04-8024-6dfe0a1cdb66" containerName="payload-pre-processing" Nov 04 01:44:21.621282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620925 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e763af-df4c-41fe-8fc5-d8677f2b4f40" containerName="istio-proxy" Nov 04 01:44:21.621282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.620931 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c850f8-6a92-4cec-98b1-a82f9d2785d2" containerName="payload-processing" Nov 04 01:44:21.623588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.623571 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:21.629025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.629001 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zklsh\"" Nov 04 01:44:21.636970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.636946 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb"] Nov 04 01:44:21.673768 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.673743 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqg5\" (UniqueName: \"kubernetes.io/projected/f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87-kube-api-access-8vqg5\") pod \"kuadrant-operator-controller-manager-678d87766f-rztrb\" (UID: \"f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:21.673921 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.673903 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rztrb\" (UID: \"f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:21.775268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.775242 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rztrb\" (UID: \"f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:21.775375 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.775288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqg5\" (UniqueName: \"kubernetes.io/projected/f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87-kube-api-access-8vqg5\") pod \"kuadrant-operator-controller-manager-678d87766f-rztrb\" (UID: \"f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:21.775632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.775611 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rztrb\" (UID: \"f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:21.784601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.784578 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqg5\" (UniqueName: \"kubernetes.io/projected/f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87-kube-api-access-8vqg5\") pod \"kuadrant-operator-controller-manager-678d87766f-rztrb\" (UID: \"f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:21.936821 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:21.936758 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:22.248417 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:22.248391 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb"] Nov 04 01:44:22.250545 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:44:22.250518 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0dabfeb_1af7_4df9_9e2b_b96aa0b04d87.slice/crio-1f6a4943692fec34dd0a7b294ceb4b8be4bdf532beb384bd511e6e9c7f2d2661 WatchSource:0}: Error finding container 1f6a4943692fec34dd0a7b294ceb4b8be4bdf532beb384bd511e6e9c7f2d2661: Status 404 returned error can't find the container with id 1f6a4943692fec34dd0a7b294ceb4b8be4bdf532beb384bd511e6e9c7f2d2661 Nov 04 01:44:23.145459 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:23.145421 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" event={"ID":"f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87","Type":"ContainerStarted","Data":"39e0454fc2ac5786b72f3b9adadf6c91d1ea0c57f29835933b52173961594723"} Nov 04 01:44:23.145855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:23.145482 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" event={"ID":"f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87","Type":"ContainerStarted","Data":"1f6a4943692fec34dd0a7b294ceb4b8be4bdf532beb384bd511e6e9c7f2d2661"} Nov 04 01:44:23.145855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:23.145562 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:44:23.167709 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:23.167601 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" podStartSLOduration=2.1675886269999998 podStartE2EDuration="2.167588627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:44:23.165231008 +0000 UTC m=+2802.136979879" watchObservedRunningTime="2026-11-04 01:44:23.167588627 +0000 UTC m=+2802.139337507" Nov 04 01:44:34.105005 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:44:34.104965 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:44:34.154143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:44:34.154090 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rztrb" Nov 04 01:45:01.534847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.534809 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt"] Nov 04 01:45:01.535313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.535273 5051 cpu_manager.go:414] "containerMap: removing container" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" Nov 04 01:45:01.535313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.535284 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" Nov 04 01:45:01.535397 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.535376 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c49724a-9d0f-42ec-86c8-cf8f7229f38a" containerName="manager" Nov 04 01:45:01.540063 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.539893 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.544139 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.544091 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:45:01.545338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.545199 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:45:01.545338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.545237 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:45:01.545338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.545275 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:45:01.548160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.548106 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt"] Nov 04 01:45:01.702352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.702310 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.702528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.702362 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbw4k\" (UniqueName: \"kubernetes.io/projected/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-kube-api-access-hbw4k\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.702528 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.702472 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.804010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.803929 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.804010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.803969 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbw4k\" (UniqueName: \"kubernetes.io/projected/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-kube-api-access-hbw4k\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.804234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.804017 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.804627 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.804595 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.806720 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.806698 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.813772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.813745 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbw4k\" (UniqueName: \"kubernetes.io/projected/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-kube-api-access-hbw4k\") pod \"maas-api-revoke-sub-d2bc93bc-2b6vt\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:01.857280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:01.857251 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:02.146782 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:02.146755 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt"] Nov 04 01:45:02.149441 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:45:02.149410 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b80638f_db8e_4fe0_9f10_ad9dd670b3ed.slice/crio-5d59c26832e3f6631b2bb5ba93468515aaac7588defcce9a7e38204b993979cb WatchSource:0}: Error finding container 5d59c26832e3f6631b2bb5ba93468515aaac7588defcce9a7e38204b993979cb: Status 404 returned error can't find the container with id 5d59c26832e3f6631b2bb5ba93468515aaac7588defcce9a7e38204b993979cb Nov 04 01:45:02.488953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:02.488917 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" event={"ID":"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed","Type":"ContainerStarted","Data":"ea7eef5bdee284f89fbb5250bcb0055ab4bcf7ac072b21bdc078d970bea6bd89"} Nov 04 01:45:02.489087 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:02.488986 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" event={"ID":"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed","Type":"ContainerStarted","Data":"5d59c26832e3f6631b2bb5ba93468515aaac7588defcce9a7e38204b993979cb"} Nov 04 01:45:02.507480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:02.507428 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" podStartSLOduration=1.507411595 podStartE2EDuration="1.507411595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:02.506961278 +0000 UTC m=+2841.478710148" watchObservedRunningTime="2026-11-04 01:45:02.507411595 +0000 UTC m=+2841.479160469" Nov 04 01:45:03.496631 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:03.496589 5051 generic.go:356] "Generic (PLEG): container finished" podID="2b80638f-db8e-4fe0-9f10-ad9dd670b3ed" containerID="ea7eef5bdee284f89fbb5250bcb0055ab4bcf7ac072b21bdc078d970bea6bd89" exitCode=0 Nov 04 01:45:03.497044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:03.496633 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" event={"ID":"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed","Type":"ContainerDied","Data":"ea7eef5bdee284f89fbb5250bcb0055ab4bcf7ac072b21bdc078d970bea6bd89"} Nov 04 01:45:04.106718 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:45:04.106677 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:45:04.654956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.654929 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:04.837178 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.837076 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-kube-api-access-hbw4k\" (UniqueName: \"kubernetes.io/projected/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-kube-api-access-hbw4k\") pod \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " Nov 04 01:45:04.837352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.837256 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-openshift-ca\") pod \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " Nov 04 01:45:04.837416 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.837385 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-xks-ca\") pod \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\" (UID: \"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed\") " Nov 04 01:45:04.837668 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.837637 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-openshift-ca" pod "2b80638f-db8e-4fe0-9f10-ad9dd670b3ed" (UID: "2b80638f-db8e-4fe0-9f10-ad9dd670b3ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:45:04.837789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.837745 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:04.839403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.839374 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-kube-api-access-hbw4k" pod "2b80638f-db8e-4fe0-9f10-ad9dd670b3ed" (UID: "2b80638f-db8e-4fe0-9f10-ad9dd670b3ed"). InnerVolumeSpecName "kube-api-access-hbw4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:45:04.839510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.839471 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-xks-ca" pod "2b80638f-db8e-4fe0-9f10-ad9dd670b3ed" (UID: "2b80638f-db8e-4fe0-9f10-ad9dd670b3ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:45:04.938778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.938745 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:04.938778 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:04.938776 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbw4k\" (UniqueName: \"kubernetes.io/projected/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed-kube-api-access-hbw4k\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:45:05.510313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:05.510273 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" event={"ID":"2b80638f-db8e-4fe0-9f10-ad9dd670b3ed","Type":"ContainerDied","Data":"5d59c26832e3f6631b2bb5ba93468515aaac7588defcce9a7e38204b993979cb"} Nov 04 01:45:05.510313 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:05.510295 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt" Nov 04 01:45:05.510535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:05.510333 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d59c26832e3f6631b2bb5ba93468515aaac7588defcce9a7e38204b993979cb" Nov 04 01:45:13.377148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.377099 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs"] Nov 04 01:45:13.377659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.377564 5051 cpu_manager.go:414] "containerMap: removing container" podUID="2b80638f-db8e-4fe0-9f10-ad9dd670b3ed" containerName="revoke-keys" Nov 04 01:45:13.377659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.377577 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b80638f-db8e-4fe0-9f10-ad9dd670b3ed" containerName="revoke-keys" Nov 04 01:45:13.377659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.377658 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b80638f-db8e-4fe0-9f10-ad9dd670b3ed" containerName="revoke-keys" Nov 04 01:45:13.380791 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.380763 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.390531 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.390392 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-2ba03d-openshift-default-dockercfg-wqd4c\"" Nov 04 01:45:13.403259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.403234 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs"] Nov 04 01:45:13.413912 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.413886 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qqc\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-kube-api-access-56qqc\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.414025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.413929 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-podinfo\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.414025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.413957 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-envoy\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.414025 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.413996 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-data\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.414207 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.414103 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-token\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.414300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.414277 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-socket\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.414360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.414321 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-certs\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.414399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.414367 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd36e15-81ce-4e9b-81ab-70444170a621-istiod-ca-cert\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.414435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.414403 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-credential-socket\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.516394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.516351 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-certs\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.517601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.517573 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-certs\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.517958 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.517929 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd36e15-81ce-4e9b-81ab-70444170a621-istiod-ca-cert\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.518062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.518031 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-credential-socket\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.518210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.518093 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qqc\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-kube-api-access-56qqc\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.518210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.518158 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-podinfo\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.518210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.518192 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-envoy\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.518394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.518222 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-data\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.518394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.518317 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-token\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.518533 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.518442 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-socket\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.519629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.519600 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-socket\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.519758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.519608 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-data\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.519758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.519612 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-credential-socket\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.520379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.520181 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd36e15-81ce-4e9b-81ab-70444170a621-istiod-ca-cert\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.522928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.522903 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-envoy\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.523059 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.523038 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-podinfo\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.528234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.528209 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qqc\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-kube-api-access-56qqc\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.528342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.528326 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-token\") pod \"e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.697426 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.697347 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:13.982452 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.982337 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs"] Nov 04 01:45:13.985189 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:45:13.985159 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd36e15_81ce_4e9b_81ab_70444170a621.slice/crio-0a8537b2a10f87774ea20fd79324d5b2b3ef2490036087ab9c78688a08374e95 WatchSource:0}: Error finding container 0a8537b2a10f87774ea20fd79324d5b2b3ef2490036087ab9c78688a08374e95: Status 404 returned error can't find the container with id 0a8537b2a10f87774ea20fd79324d5b2b3ef2490036087ab9c78688a08374e95 Nov 04 01:45:13.987468 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.987435 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:45:13.987571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.987504 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:45:13.987571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:13.987535 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:45:14.560169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:14.560131 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" event={"ID":"7dd36e15-81ce-4e9b-81ab-70444170a621","Type":"ContainerStarted","Data":"7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e"} Nov 04 01:45:14.560551 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:14.560194 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" event={"ID":"7dd36e15-81ce-4e9b-81ab-70444170a621","Type":"ContainerStarted","Data":"0a8537b2a10f87774ea20fd79324d5b2b3ef2490036087ab9c78688a08374e95"} Nov 04 01:45:14.586322 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:14.586267 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" podStartSLOduration=1.58625245 podStartE2EDuration="1.58625245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:14.583351608 +0000 UTC m=+2853.555100479" watchObservedRunningTime="2026-11-04 01:45:14.58625245 +0000 UTC m=+2853.558001318" Nov 04 01:45:14.698469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:14.698409 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:15.703247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:15.703214 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:16.570565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:16.570524 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:16.571471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:16.571448 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:18.076930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.076896 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx"] Nov 04 01:45:18.081101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.081075 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:18.086626 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.086607 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fdd9d6a3-dockercfg-2m4l8\"" Nov 04 01:45:18.091387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.091362 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2ba03d\"" Nov 04 01:45:18.094742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.094716 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx"] Nov 04 01:45:18.111569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.111540 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c"] Nov 04 01:45:18.115317 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.115300 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:18.127372 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.127341 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c"] Nov 04 01:45:18.165799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.165762 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246ng\" (UniqueName: \"kubernetes.io/projected/8894915f-999d-4b3d-8742-c459eb54b7e2-kube-api-access-246ng\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx\" (UID: \"8894915f-999d-4b3d-8742-c459eb54b7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:18.165953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.165880 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8894915f-999d-4b3d-8742-c459eb54b7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx\" (UID: \"8894915f-999d-4b3d-8742-c459eb54b7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:18.165953 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.165948 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026294f-4429-42d3-8b8e-612f1f26b9c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c\" (UID: \"a026294f-4429-42d3-8b8e-612f1f26b9c9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:18.166106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.165978 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb5bn\" (UniqueName: \"kubernetes.io/projected/a026294f-4429-42d3-8b8e-612f1f26b9c9-kube-api-access-gb5bn\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c\" (UID: \"a026294f-4429-42d3-8b8e-612f1f26b9c9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:18.267282 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.267244 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8894915f-999d-4b3d-8742-c459eb54b7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx\" (UID: \"8894915f-999d-4b3d-8742-c459eb54b7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:18.267494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.267313 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026294f-4429-42d3-8b8e-612f1f26b9c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c\" (UID: \"a026294f-4429-42d3-8b8e-612f1f26b9c9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:18.267494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.267341 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb5bn\" (UniqueName: \"kubernetes.io/projected/a026294f-4429-42d3-8b8e-612f1f26b9c9-kube-api-access-gb5bn\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c\" (UID: \"a026294f-4429-42d3-8b8e-612f1f26b9c9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:18.267494 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.267412 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246ng\" (UniqueName: \"kubernetes.io/projected/8894915f-999d-4b3d-8742-c459eb54b7e2-kube-api-access-246ng\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx\" (UID: \"8894915f-999d-4b3d-8742-c459eb54b7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:18.268316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.268285 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8894915f-999d-4b3d-8742-c459eb54b7e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx\" (UID: \"8894915f-999d-4b3d-8742-c459eb54b7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:18.268407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.268285 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026294f-4429-42d3-8b8e-612f1f26b9c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c\" (UID: \"a026294f-4429-42d3-8b8e-612f1f26b9c9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:18.276325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.276292 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb5bn\" (UniqueName: \"kubernetes.io/projected/a026294f-4429-42d3-8b8e-612f1f26b9c9-kube-api-access-gb5bn\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c\" (UID: \"a026294f-4429-42d3-8b8e-612f1f26b9c9\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:18.276635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.276609 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-246ng\" (UniqueName: \"kubernetes.io/projected/8894915f-999d-4b3d-8742-c459eb54b7e2-kube-api-access-246ng\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx\" (UID: \"8894915f-999d-4b3d-8742-c459eb54b7e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:18.397565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.397496 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:18.432576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.432543 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:18.685964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.685931 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx"] Nov 04 01:45:18.688662 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:45:18.688631 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8894915f_999d_4b3d_8742_c459eb54b7e2.slice/crio-8bcb1131fa54ea381e44a03cd9e6d820fcf319755a72a6affcc61b1284d2da1a WatchSource:0}: Error finding container 8bcb1131fa54ea381e44a03cd9e6d820fcf319755a72a6affcc61b1284d2da1a: Status 404 returned error can't find the container with id 8bcb1131fa54ea381e44a03cd9e6d820fcf319755a72a6affcc61b1284d2da1a Nov 04 01:45:18.728402 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:18.728367 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c"] Nov 04 01:45:18.746504 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:45:18.746474 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda026294f_4429_42d3_8b8e_612f1f26b9c9.slice/crio-4070c105ba89eed7828a430809ff10768da85c66f0ea760c8ef86bb9327ee3bc WatchSource:0}: Error finding container 4070c105ba89eed7828a430809ff10768da85c66f0ea760c8ef86bb9327ee3bc: Status 404 returned error can't find the container with id 4070c105ba89eed7828a430809ff10768da85c66f0ea760c8ef86bb9327ee3bc Nov 04 01:45:19.604430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:19.604370 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" event={"ID":"8894915f-999d-4b3d-8742-c459eb54b7e2","Type":"ContainerStarted","Data":"bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1"} Nov 04 01:45:19.604870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:19.604458 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" event={"ID":"8894915f-999d-4b3d-8742-c459eb54b7e2","Type":"ContainerStarted","Data":"8bcb1131fa54ea381e44a03cd9e6d820fcf319755a72a6affcc61b1284d2da1a"} Nov 04 01:45:19.604870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:19.604607 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:19.606450 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:19.606418 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" event={"ID":"a026294f-4429-42d3-8b8e-612f1f26b9c9","Type":"ContainerStarted","Data":"62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434"} Nov 04 01:45:19.606579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:19.606479 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:19.606579 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:19.606502 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" event={"ID":"a026294f-4429-42d3-8b8e-612f1f26b9c9","Type":"ContainerStarted","Data":"4070c105ba89eed7828a430809ff10768da85c66f0ea760c8ef86bb9327ee3bc"} Nov 04 01:45:19.634164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:19.634078 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" podStartSLOduration=1.6340589589999999 podStartE2EDuration="1.634058959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:19.630788632 +0000 UTC m=+2858.602537504" watchObservedRunningTime="2026-11-04 01:45:19.634058959 +0000 UTC m=+2858.605807835" Nov 04 01:45:19.654471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:19.654412 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" podStartSLOduration=1.654393837 podStartE2EDuration="1.654393837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:45:19.653526267 +0000 UTC m=+2858.625275140" watchObservedRunningTime="2026-11-04 01:45:19.654393837 +0000 UTC m=+2858.626142709" Nov 04 01:45:30.615853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:30.615812 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:45:30.618218 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:30.615905 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:45:31.572636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:31.572596 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 04 01:45:31.572838 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:31.572662 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 04 01:45:31.704449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:31.704409 5051 generic.go:356] "Generic (PLEG): container finished" podID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerID="7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e" exitCode=0 Nov 04 01:45:31.704959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:31.704456 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" event={"ID":"7dd36e15-81ce-4e9b-81ab-70444170a621","Type":"ContainerDied","Data":"7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e"} Nov 04 01:45:31.704959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:31.704943 5051 scope.go:122] "RemoveContainer" containerID="7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e" Nov 04 01:45:31.706204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:31.706171 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:45:31.706306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:31.706242 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:45:31.706306 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:31.706273 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:45:32.714182 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:32.714144 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" event={"ID":"7dd36e15-81ce-4e9b-81ab-70444170a621","Type":"ContainerStarted","Data":"393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c"} Nov 04 01:45:33.698115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:33.698076 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:33.698369 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:33.698336 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 04 01:45:33.698436 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:33.698385 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 04 01:45:34.112522 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:45:34.112472 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:45:34.698137 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:34.698092 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 04 01:45:34.698357 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:34.698158 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 04 01:45:35.698484 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:35.698435 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 04 01:45:35.699019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:35.698503 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 04 01:45:36.699585 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:36.699539 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 04 01:45:36.700100 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:36.699605 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 04 01:45:37.698325 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:37.698278 5051 patch_prober.go:29] interesting pod/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Nov 04 01:45:37.698546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:37.698352 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Nov 04 01:45:38.702027 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:38.701996 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:38.702487 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:38.702385 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:38.703026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:38.703006 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:45:50.069754 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:45:50.069721 5051 scope.go:122] "RemoveContainer" containerID="f41727a6be62b10c9a95f6f8969d797b433dae9125cae8cacf7ec570d2dd8558" Nov 04 01:46:03.812259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.812221 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4"] Nov 04 01:46:03.817291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.817271 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:03.822189 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.822159 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:46:03.822316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.822205 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:46:03.822365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.822355 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:46:03.822462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.822443 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:46:03.824813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.824698 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4"] Nov 04 01:46:03.917107 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.917071 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txk62\" (UniqueName: \"kubernetes.io/projected/0f313598-269b-4ac5-b6de-c141f56f20b5-kube-api-access-txk62\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:03.917315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.917142 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:03.917315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:03.917283 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:04.017863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.017824 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:04.018075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.017887 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txk62\" (UniqueName: \"kubernetes.io/projected/0f313598-269b-4ac5-b6de-c141f56f20b5-kube-api-access-txk62\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:04.018075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.017915 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:04.018764 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.018734 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:04.020361 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.020336 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:04.026547 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.026522 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txk62\" (UniqueName: \"kubernetes.io/projected/0f313598-269b-4ac5-b6de-c141f56f20b5-kube-api-access-txk62\") pod \"maas-api-revoke-sub-b35766fe-jskt4\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:04.111315 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:46:04.111215 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:46:04.132956 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.132927 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:04.413930 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.413904 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4"] Nov 04 01:46:04.416444 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:46:04.416408 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f313598_269b_4ac5_b6de_c141f56f20b5.slice/crio-8c78b89f383e730bc719532ce96f2149a8ef2012857b2284adc944fea9221198 WatchSource:0}: Error finding container 8c78b89f383e730bc719532ce96f2149a8ef2012857b2284adc944fea9221198: Status 404 returned error can't find the container with id 8c78b89f383e730bc719532ce96f2149a8ef2012857b2284adc944fea9221198 Nov 04 01:46:04.909152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.909105 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" event={"ID":"0f313598-269b-4ac5-b6de-c141f56f20b5","Type":"ContainerStarted","Data":"233380d7515df37111e2633bca302a0c5863ed1aa530e8b2fcf118a7021404ef"} Nov 04 01:46:04.909569 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.909171 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" event={"ID":"0f313598-269b-4ac5-b6de-c141f56f20b5","Type":"ContainerStarted","Data":"8c78b89f383e730bc719532ce96f2149a8ef2012857b2284adc944fea9221198"} Nov 04 01:46:04.930966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:04.930901 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" podStartSLOduration=1.930881028 podStartE2EDuration="1.930881028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:04.928665352 +0000 UTC m=+2903.900414219" watchObservedRunningTime="2026-11-04 01:46:04.930881028 +0000 UTC m=+2903.902629900" Nov 04 01:46:05.916286 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:05.916251 5051 generic.go:356] "Generic (PLEG): container finished" podID="0f313598-269b-4ac5-b6de-c141f56f20b5" containerID="233380d7515df37111e2633bca302a0c5863ed1aa530e8b2fcf118a7021404ef" exitCode=0 Nov 04 01:46:05.916802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:05.916337 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" event={"ID":"0f313598-269b-4ac5-b6de-c141f56f20b5","Type":"ContainerDied","Data":"233380d7515df37111e2633bca302a0c5863ed1aa530e8b2fcf118a7021404ef"} Nov 04 01:46:07.063652 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.063622 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:07.154211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.154171 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f313598-269b-4ac5-b6de-c141f56f20b5-kube-api-access-txk62\" (UniqueName: \"kubernetes.io/projected/0f313598-269b-4ac5-b6de-c141f56f20b5-kube-api-access-txk62\") pod \"0f313598-269b-4ac5-b6de-c141f56f20b5\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " Nov 04 01:46:07.154211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.154217 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-xks-ca\") pod \"0f313598-269b-4ac5-b6de-c141f56f20b5\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " Nov 04 01:46:07.154469 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.154317 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-openshift-ca\") pod \"0f313598-269b-4ac5-b6de-c141f56f20b5\" (UID: \"0f313598-269b-4ac5-b6de-c141f56f20b5\") " Nov 04 01:46:07.154728 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.154701 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-openshift-ca" pod "0f313598-269b-4ac5-b6de-c141f56f20b5" (UID: "0f313598-269b-4ac5-b6de-c141f56f20b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:07.156944 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.156916 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-xks-ca" pod "0f313598-269b-4ac5-b6de-c141f56f20b5" (UID: "0f313598-269b-4ac5-b6de-c141f56f20b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:46:07.157085 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.157031 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f313598-269b-4ac5-b6de-c141f56f20b5-kube-api-access-txk62" pod "0f313598-269b-4ac5-b6de-c141f56f20b5" (UID: "0f313598-269b-4ac5-b6de-c141f56f20b5"). InnerVolumeSpecName "kube-api-access-txk62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:07.255259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.255221 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:07.255259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.255253 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txk62\" (UniqueName: \"kubernetes.io/projected/0f313598-269b-4ac5-b6de-c141f56f20b5-kube-api-access-txk62\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:07.255259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.255264 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f313598-269b-4ac5-b6de-c141f56f20b5-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:07.938523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.938490 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" Nov 04 01:46:07.938706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.938486 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4" event={"ID":"0f313598-269b-4ac5-b6de-c141f56f20b5","Type":"ContainerDied","Data":"8c78b89f383e730bc719532ce96f2149a8ef2012857b2284adc944fea9221198"} Nov 04 01:46:07.938706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:07.938636 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c78b89f383e730bc719532ce96f2149a8ef2012857b2284adc944fea9221198" Nov 04 01:46:21.015594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:21.015557 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c"] Nov 04 01:46:21.028706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:21.028675 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx"] Nov 04 01:46:26.016485 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.016375 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" podUID="a026294f-4429-42d3-8b8e-612f1f26b9c9" containerName="payload-processing" containerID="cri-o://62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434" gracePeriod=25 Nov 04 01:46:26.029932 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.029897 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" podUID="8894915f-999d-4b3d-8742-c459eb54b7e2" containerName="payload-pre-processing" containerID="cri-o://bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1" gracePeriod=25 Nov 04 01:46:26.304755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.304724 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:46:26.328594 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.328569 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:46:26.349426 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.349392 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a026294f-4429-42d3-8b8e-612f1f26b9c9-kube-api-access-gb5bn\" (UniqueName: \"kubernetes.io/projected/a026294f-4429-42d3-8b8e-612f1f26b9c9-kube-api-access-gb5bn\") pod \"a026294f-4429-42d3-8b8e-612f1f26b9c9\" (UID: \"a026294f-4429-42d3-8b8e-612f1f26b9c9\") " Nov 04 01:46:26.349655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.349604 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a026294f-4429-42d3-8b8e-612f1f26b9c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026294f-4429-42d3-8b8e-612f1f26b9c9-plugins-config-volume\") pod \"a026294f-4429-42d3-8b8e-612f1f26b9c9\" (UID: \"a026294f-4429-42d3-8b8e-612f1f26b9c9\") " Nov 04 01:46:26.349994 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.349947 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a026294f-4429-42d3-8b8e-612f1f26b9c9-plugins-config-volume" pod "a026294f-4429-42d3-8b8e-612f1f26b9c9" (UID: "a026294f-4429-42d3-8b8e-612f1f26b9c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:26.350151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.350087 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026294f-4429-42d3-8b8e-612f1f26b9c9-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:26.354398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.354368 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a026294f-4429-42d3-8b8e-612f1f26b9c9-kube-api-access-gb5bn" pod "a026294f-4429-42d3-8b8e-612f1f26b9c9" (UID: "a026294f-4429-42d3-8b8e-612f1f26b9c9"). InnerVolumeSpecName "kube-api-access-gb5bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:26.451018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.450984 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8894915f-999d-4b3d-8742-c459eb54b7e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8894915f-999d-4b3d-8742-c459eb54b7e2-plugins-config-volume\") pod \"8894915f-999d-4b3d-8742-c459eb54b7e2\" (UID: \"8894915f-999d-4b3d-8742-c459eb54b7e2\") " Nov 04 01:46:26.451225 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.451026 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8894915f-999d-4b3d-8742-c459eb54b7e2-kube-api-access-246ng\" (UniqueName: \"kubernetes.io/projected/8894915f-999d-4b3d-8742-c459eb54b7e2-kube-api-access-246ng\") pod \"8894915f-999d-4b3d-8742-c459eb54b7e2\" (UID: \"8894915f-999d-4b3d-8742-c459eb54b7e2\") " Nov 04 01:46:26.451295 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.451283 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb5bn\" (UniqueName: \"kubernetes.io/projected/a026294f-4429-42d3-8b8e-612f1f26b9c9-kube-api-access-gb5bn\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:26.451405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.451382 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8894915f-999d-4b3d-8742-c459eb54b7e2-plugins-config-volume" pod "8894915f-999d-4b3d-8742-c459eb54b7e2" (UID: "8894915f-999d-4b3d-8742-c459eb54b7e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:46:26.453246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.453221 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8894915f-999d-4b3d-8742-c459eb54b7e2-kube-api-access-246ng" pod "8894915f-999d-4b3d-8742-c459eb54b7e2" (UID: "8894915f-999d-4b3d-8742-c459eb54b7e2"). InnerVolumeSpecName "kube-api-access-246ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:46:26.552445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.552378 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8894915f-999d-4b3d-8742-c459eb54b7e2-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:26.552445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:26.552402 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246ng\" (UniqueName: \"kubernetes.io/projected/8894915f-999d-4b3d-8742-c459eb54b7e2-kube-api-access-246ng\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:46:27.056571 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.056537 5051 generic.go:356] "Generic (PLEG): container finished" podID="8894915f-999d-4b3d-8742-c459eb54b7e2" containerID="bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1" exitCode=0 Nov 04 01:46:27.056985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.056578 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" Nov 04 01:46:27.056985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.056650 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" event={"ID":"8894915f-999d-4b3d-8742-c459eb54b7e2","Type":"ContainerDied","Data":"bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1"} Nov 04 01:46:27.056985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.056705 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx" event={"ID":"8894915f-999d-4b3d-8742-c459eb54b7e2","Type":"ContainerDied","Data":"8bcb1131fa54ea381e44a03cd9e6d820fcf319755a72a6affcc61b1284d2da1a"} Nov 04 01:46:27.056985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.056735 5051 scope.go:122] "RemoveContainer" containerID="bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1" Nov 04 01:46:27.058522 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.058498 5051 generic.go:356] "Generic (PLEG): container finished" podID="a026294f-4429-42d3-8b8e-612f1f26b9c9" containerID="62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434" exitCode=0 Nov 04 01:46:27.058601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.058537 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" Nov 04 01:46:27.058601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.058582 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" event={"ID":"a026294f-4429-42d3-8b8e-612f1f26b9c9","Type":"ContainerDied","Data":"62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434"} Nov 04 01:46:27.058706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.058632 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c" event={"ID":"a026294f-4429-42d3-8b8e-612f1f26b9c9","Type":"ContainerDied","Data":"4070c105ba89eed7828a430809ff10768da85c66f0ea760c8ef86bb9327ee3bc"} Nov 04 01:46:27.075352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.075315 5051 scope.go:122] "RemoveContainer" containerID="bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1" Nov 04 01:46:27.075969 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:46:27.075902 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1\": container with ID starting with bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1 not found: ID does not exist" containerID="bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1" Nov 04 01:46:27.075969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.075945 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1"} err="failed to get container status \"bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1\": rpc error: code = NotFound desc = could not find container \"bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1\": container with ID starting with bd7856a34321fab28ff284e06abdf5dca24df3eda34f2104ced8fccd605921b1 not found: ID does not exist" Nov 04 01:46:27.075969 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.075969 5051 scope.go:122] "RemoveContainer" containerID="62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434" Nov 04 01:46:27.087920 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.087902 5051 scope.go:122] "RemoveContainer" containerID="62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434" Nov 04 01:46:27.088199 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:46:27.088178 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434\": container with ID starting with 62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434 not found: ID does not exist" containerID="62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434" Nov 04 01:46:27.088251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.088207 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434"} err="failed to get container status \"62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434\": rpc error: code = NotFound desc = could not find container \"62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434\": container with ID starting with 62d0d9f5d484c5adc9e882025401ef92d3c65c6fa78096ff64035ebfde4c0434 not found: ID does not exist" Nov 04 01:46:27.092421 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.092391 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c"] Nov 04 01:46:27.094449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.094430 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bfz7c"] Nov 04 01:46:27.104047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.104026 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx"] Nov 04 01:46:27.107913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.107892 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9d58jx"] Nov 04 01:46:27.187959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.187928 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8894915f-999d-4b3d-8742-c459eb54b7e2" path="/var/lib/kubelet/pods/8894915f-999d-4b3d-8742-c459eb54b7e2/volumes" Nov 04 01:46:27.188373 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:27.188359 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a026294f-4429-42d3-8b8e-612f1f26b9c9" path="/var/lib/kubelet/pods/a026294f-4429-42d3-8b8e-612f1f26b9c9/volumes" Nov 04 01:46:34.105486 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:46:34.105444 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:46:53.252265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.252222 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97"] Nov 04 01:46:53.252908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.252889 5051 cpu_manager.go:414] "containerMap: removing container" podUID="0f313598-269b-4ac5-b6de-c141f56f20b5" containerName="revoke-keys" Nov 04 01:46:53.252959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.252910 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f313598-269b-4ac5-b6de-c141f56f20b5" containerName="revoke-keys" Nov 04 01:46:53.252959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.252940 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a026294f-4429-42d3-8b8e-612f1f26b9c9" containerName="payload-processing" Nov 04 01:46:53.252959 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.252948 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a026294f-4429-42d3-8b8e-612f1f26b9c9" containerName="payload-processing" Nov 04 01:46:53.253065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.252986 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8894915f-999d-4b3d-8742-c459eb54b7e2" containerName="payload-pre-processing" Nov 04 01:46:53.253065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.252995 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8894915f-999d-4b3d-8742-c459eb54b7e2" containerName="payload-pre-processing" Nov 04 01:46:53.253157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.253114 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a026294f-4429-42d3-8b8e-612f1f26b9c9" containerName="payload-processing" Nov 04 01:46:53.253157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.253150 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8894915f-999d-4b3d-8742-c459eb54b7e2" containerName="payload-pre-processing" Nov 04 01:46:53.253247 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.253163 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f313598-269b-4ac5-b6de-c141f56f20b5" containerName="revoke-keys" Nov 04 01:46:53.258014 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.257993 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp"] Nov 04 01:46:53.258173 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.258151 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:53.265303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.265211 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:53.268473 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.268448 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-fdd9d6a3-dockercfg-lp5dk\"" Nov 04 01:46:53.271342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.271317 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2ba03d\"" Nov 04 01:46:53.272231 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.272203 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp"] Nov 04 01:46:53.274490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.274423 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97"] Nov 04 01:46:53.308765 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.308735 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d96d\" (UniqueName: \"kubernetes.io/projected/56291e58-e3e7-47a7-844d-3a4b13ba7d00-kube-api-access-2d96d\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97\" (UID: \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:53.308895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.308770 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84f2g\" (UniqueName: \"kubernetes.io/projected/44ef8d77-d370-4d39-9812-73032fbed1ee-kube-api-access-84f2g\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp\" (UID: \"44ef8d77-d370-4d39-9812-73032fbed1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:53.308895 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.308809 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56291e58-e3e7-47a7-844d-3a4b13ba7d00-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97\" (UID: \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:53.308985 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.308928 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ef8d77-d370-4d39-9812-73032fbed1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp\" (UID: \"44ef8d77-d370-4d39-9812-73032fbed1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:53.409882 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.409845 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d96d\" (UniqueName: \"kubernetes.io/projected/56291e58-e3e7-47a7-844d-3a4b13ba7d00-kube-api-access-2d96d\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97\" (UID: \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:53.410077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.409891 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84f2g\" (UniqueName: \"kubernetes.io/projected/44ef8d77-d370-4d39-9812-73032fbed1ee-kube-api-access-84f2g\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp\" (UID: \"44ef8d77-d370-4d39-9812-73032fbed1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:53.410077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.409929 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56291e58-e3e7-47a7-844d-3a4b13ba7d00-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97\" (UID: \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:53.410077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.410029 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ef8d77-d370-4d39-9812-73032fbed1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp\" (UID: \"44ef8d77-d370-4d39-9812-73032fbed1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:53.410816 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.410790 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56291e58-e3e7-47a7-844d-3a4b13ba7d00-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97\" (UID: \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:53.410916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.410790 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ef8d77-d370-4d39-9812-73032fbed1ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp\" (UID: \"44ef8d77-d370-4d39-9812-73032fbed1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:53.424034 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.423998 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84f2g\" (UniqueName: \"kubernetes.io/projected/44ef8d77-d370-4d39-9812-73032fbed1ee-kube-api-access-84f2g\") pod \"payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp\" (UID: \"44ef8d77-d370-4d39-9812-73032fbed1ee\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:53.424158 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.424106 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d96d\" (UniqueName: \"kubernetes.io/projected/56291e58-e3e7-47a7-844d-3a4b13ba7d00-kube-api-access-2d96d\") pod \"payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97\" (UID: \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:53.575172 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.575075 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:53.584933 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.584905 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:53.864197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.864094 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97"] Nov 04 01:46:53.866643 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:46:53.866613 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56291e58_e3e7_47a7_844d_3a4b13ba7d00.slice/crio-0931f5cc91a336d1fee80327b98fc4c04bc3e14d91a158fa235d0624eb8e16ac WatchSource:0}: Error finding container 0931f5cc91a336d1fee80327b98fc4c04bc3e14d91a158fa235d0624eb8e16ac: Status 404 returned error can't find the container with id 0931f5cc91a336d1fee80327b98fc4c04bc3e14d91a158fa235d0624eb8e16ac Nov 04 01:46:53.878327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:53.878037 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp"] Nov 04 01:46:53.880452 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:46:53.880424 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ef8d77_d370_4d39_9812_73032fbed1ee.slice/crio-fba5024d2fe1a10ba9346d16dfca01c1f4fbfd23fa4e9580bf990d7287d08e93 WatchSource:0}: Error finding container fba5024d2fe1a10ba9346d16dfca01c1f4fbfd23fa4e9580bf990d7287d08e93: Status 404 returned error can't find the container with id fba5024d2fe1a10ba9346d16dfca01c1f4fbfd23fa4e9580bf990d7287d08e93 Nov 04 01:46:54.208412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:54.208318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" event={"ID":"44ef8d77-d370-4d39-9812-73032fbed1ee","Type":"ContainerStarted","Data":"904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638"} Nov 04 01:46:54.208412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:54.208391 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:46:54.208656 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:54.208420 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" event={"ID":"44ef8d77-d370-4d39-9812-73032fbed1ee","Type":"ContainerStarted","Data":"fba5024d2fe1a10ba9346d16dfca01c1f4fbfd23fa4e9580bf990d7287d08e93"} Nov 04 01:46:54.210006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:54.209975 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" event={"ID":"56291e58-e3e7-47a7-844d-3a4b13ba7d00","Type":"ContainerStarted","Data":"46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d"} Nov 04 01:46:54.210165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:54.210021 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" event={"ID":"56291e58-e3e7-47a7-844d-3a4b13ba7d00","Type":"ContainerStarted","Data":"0931f5cc91a336d1fee80327b98fc4c04bc3e14d91a158fa235d0624eb8e16ac"} Nov 04 01:46:54.210165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:54.210041 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:46:54.227588 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:54.227547 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" podStartSLOduration=1.227534693 podStartE2EDuration="1.227534693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:54.225711147 +0000 UTC m=+2953.197460018" watchObservedRunningTime="2026-11-04 01:46:54.227534693 +0000 UTC m=+2953.199283561" Nov 04 01:46:54.247115 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:54.247070 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" podStartSLOduration=1.24705887 podStartE2EDuration="1.24705887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:46:54.243502149 +0000 UTC m=+2953.215251033" watchObservedRunningTime="2026-11-04 01:46:54.24705887 +0000 UTC m=+2953.218807781" Nov 04 01:46:58.032857 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:58.032818 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf"] Nov 04 01:46:58.039238 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:58.039211 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-845fbef0-wtqgf"] Nov 04 01:46:59.189909 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:46:59.189873 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79eabe5-6e35-4c26-af5b-095d4c6012cd" path="/var/lib/kubelet/pods/e79eabe5-6e35-4c26-af5b-095d4c6012cd/volumes" Nov 04 01:47:01.436908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.436875 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn"] Nov 04 01:47:01.440598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.440576 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.447211 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.447182 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:47:01.447859 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.447204 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:47:01.449257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.449235 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:47:01.449400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.449367 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:47:01.452062 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.452039 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn"] Nov 04 01:47:01.517147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.513842 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.517147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.513922 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvsvz\" (UniqueName: \"kubernetes.io/projected/8e061270-13d8-46b4-bca4-f394723a70ea-kube-api-access-qvsvz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.517147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.514059 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.615148 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.615091 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.615333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.615181 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvsvz\" (UniqueName: \"kubernetes.io/projected/8e061270-13d8-46b4-bca4-f394723a70ea-kube-api-access-qvsvz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.615333 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.615288 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.615951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.615931 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.617577 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.617556 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.624243 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.624221 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvsvz\" (UniqueName: \"kubernetes.io/projected/8e061270-13d8-46b4-bca4-f394723a70ea-kube-api-access-qvsvz\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:01.756133 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:01.756091 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:02.045387 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:02.045366 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn"] Nov 04 01:47:02.047973 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:47:02.047931 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e061270_13d8_46b4_bca4_f394723a70ea.slice/crio-e976bbcb8d6246f024d08e546f14859537a677690b01729346760873f88af3ef WatchSource:0}: Error finding container e976bbcb8d6246f024d08e546f14859537a677690b01729346760873f88af3ef: Status 404 returned error can't find the container with id e976bbcb8d6246f024d08e546f14859537a677690b01729346760873f88af3ef Nov 04 01:47:02.258188 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:02.258154 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" event={"ID":"8e061270-13d8-46b4-bca4-f394723a70ea","Type":"ContainerStarted","Data":"e976bbcb8d6246f024d08e546f14859537a677690b01729346760873f88af3ef"} Nov 04 01:47:03.265582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:03.265546 5051 generic.go:356] "Generic (PLEG): container finished" podID="8e061270-13d8-46b4-bca4-f394723a70ea" containerID="9535e3f6520adc351dae74e5c5ccebe464720d836c88be0eeddfc8df0e0cdb72" exitCode=0 Nov 04 01:47:03.265987 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:03.265632 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" event={"ID":"8e061270-13d8-46b4-bca4-f394723a70ea","Type":"ContainerDied","Data":"9535e3f6520adc351dae74e5c5ccebe464720d836c88be0eeddfc8df0e0cdb72"} Nov 04 01:47:04.105692 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:47:04.105652 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:47:04.408337 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.408313 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:04.546106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.546070 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-openshift-ca\") pod \"8e061270-13d8-46b4-bca4-f394723a70ea\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " Nov 04 01:47:04.546276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.546218 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-xks-ca\") pod \"8e061270-13d8-46b4-bca4-f394723a70ea\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " Nov 04 01:47:04.546336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.546293 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e061270-13d8-46b4-bca4-f394723a70ea-kube-api-access-qvsvz\" (UniqueName: \"kubernetes.io/projected/8e061270-13d8-46b4-bca4-f394723a70ea-kube-api-access-qvsvz\") pod \"8e061270-13d8-46b4-bca4-f394723a70ea\" (UID: \"8e061270-13d8-46b4-bca4-f394723a70ea\") " Nov 04 01:47:04.546490 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.546469 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-openshift-ca" pod "8e061270-13d8-46b4-bca4-f394723a70ea" (UID: "8e061270-13d8-46b4-bca4-f394723a70ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:04.546660 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.546641 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:04.548319 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.548295 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-xks-ca" pod "8e061270-13d8-46b4-bca4-f394723a70ea" (UID: "8e061270-13d8-46b4-bca4-f394723a70ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:47:04.548429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.548405 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e061270-13d8-46b4-bca4-f394723a70ea-kube-api-access-qvsvz" pod "8e061270-13d8-46b4-bca4-f394723a70ea" (UID: "8e061270-13d8-46b4-bca4-f394723a70ea"). InnerVolumeSpecName "kube-api-access-qvsvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:04.648691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.648602 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e061270-13d8-46b4-bca4-f394723a70ea-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:04.648691 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:04.648648 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvsvz\" (UniqueName: \"kubernetes.io/projected/8e061270-13d8-46b4-bca4-f394723a70ea-kube-api-access-qvsvz\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:05.218346 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:05.218317 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:47:05.218507 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:05.218379 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:47:05.278349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:05.278323 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" Nov 04 01:47:05.278525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:05.278317 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn" event={"ID":"8e061270-13d8-46b4-bca4-f394723a70ea","Type":"ContainerDied","Data":"e976bbcb8d6246f024d08e546f14859537a677690b01729346760873f88af3ef"} Nov 04 01:47:05.278525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:05.278462 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e976bbcb8d6246f024d08e546f14859537a677690b01729346760873f88af3ef" Nov 04 01:47:11.552715 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:11.552681 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97"] Nov 04 01:47:11.568682 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:11.568648 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp"] Nov 04 01:47:12.665529 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:12.665484 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn"] Nov 04 01:47:12.667674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:12.667651 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2ba03d-kv4qn"] Nov 04 01:47:13.025155 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:13.025090 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs"] Nov 04 01:47:13.025338 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:13.025314 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" containerID="cri-o://393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c" gracePeriod=30 Nov 04 01:47:13.188301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:13.188269 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e061270-13d8-46b4-bca4-f394723a70ea" path="/var/lib/kubelet/pods/8e061270-13d8-46b4-bca4-f394723a70ea/volumes" Nov 04 01:47:16.553757 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.553717 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" podUID="56291e58-e3e7-47a7-844d-3a4b13ba7d00" containerName="payload-processing" containerID="cri-o://46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d" gracePeriod=25 Nov 04 01:47:16.569195 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.569167 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" podUID="44ef8d77-d370-4d39-9812-73032fbed1ee" containerName="payload-pre-processing" containerID="cri-o://904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638" gracePeriod=25 Nov 04 01:47:16.863639 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.863613 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:47:16.880316 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.880291 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:47:16.975093 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.975063 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56291e58-e3e7-47a7-844d-3a4b13ba7d00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56291e58-e3e7-47a7-844d-3a4b13ba7d00-plugins-config-volume\") pod \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\" (UID: \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\") " Nov 04 01:47:16.975289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.975109 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ef8d77-d370-4d39-9812-73032fbed1ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ef8d77-d370-4d39-9812-73032fbed1ee-plugins-config-volume\") pod \"44ef8d77-d370-4d39-9812-73032fbed1ee\" (UID: \"44ef8d77-d370-4d39-9812-73032fbed1ee\") " Nov 04 01:47:16.975289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.975256 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56291e58-e3e7-47a7-844d-3a4b13ba7d00-kube-api-access-2d96d\" (UniqueName: \"kubernetes.io/projected/56291e58-e3e7-47a7-844d-3a4b13ba7d00-kube-api-access-2d96d\") pod \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\" (UID: \"56291e58-e3e7-47a7-844d-3a4b13ba7d00\") " Nov 04 01:47:16.975424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.975324 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ef8d77-d370-4d39-9812-73032fbed1ee-kube-api-access-84f2g\" (UniqueName: \"kubernetes.io/projected/44ef8d77-d370-4d39-9812-73032fbed1ee-kube-api-access-84f2g\") pod \"44ef8d77-d370-4d39-9812-73032fbed1ee\" (UID: \"44ef8d77-d370-4d39-9812-73032fbed1ee\") " Nov 04 01:47:16.975564 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.975534 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56291e58-e3e7-47a7-844d-3a4b13ba7d00-plugins-config-volume" pod "56291e58-e3e7-47a7-844d-3a4b13ba7d00" (UID: "56291e58-e3e7-47a7-844d-3a4b13ba7d00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:16.975641 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.975578 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ef8d77-d370-4d39-9812-73032fbed1ee-plugins-config-volume" pod "44ef8d77-d370-4d39-9812-73032fbed1ee" (UID: "44ef8d77-d370-4d39-9812-73032fbed1ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:16.975741 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.975714 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44ef8d77-d370-4d39-9812-73032fbed1ee-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:16.975741 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.975741 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56291e58-e3e7-47a7-844d-3a4b13ba7d00-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:16.977962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.977942 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56291e58-e3e7-47a7-844d-3a4b13ba7d00-kube-api-access-2d96d" pod "56291e58-e3e7-47a7-844d-3a4b13ba7d00" (UID: "56291e58-e3e7-47a7-844d-3a4b13ba7d00"). InnerVolumeSpecName "kube-api-access-2d96d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:16.978079 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:16.977963 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ef8d77-d370-4d39-9812-73032fbed1ee-kube-api-access-84f2g" pod "44ef8d77-d370-4d39-9812-73032fbed1ee" (UID: "44ef8d77-d370-4d39-9812-73032fbed1ee"). InnerVolumeSpecName "kube-api-access-84f2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:17.077197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.077109 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d96d\" (UniqueName: \"kubernetes.io/projected/56291e58-e3e7-47a7-844d-3a4b13ba7d00-kube-api-access-2d96d\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:17.077197 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.077153 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84f2g\" (UniqueName: \"kubernetes.io/projected/44ef8d77-d370-4d39-9812-73032fbed1ee-kube-api-access-84f2g\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:17.358662 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.358571 5051 generic.go:356] "Generic (PLEG): container finished" podID="56291e58-e3e7-47a7-844d-3a4b13ba7d00" containerID="46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d" exitCode=0 Nov 04 01:47:17.358662 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.358620 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" Nov 04 01:47:17.358919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.358654 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" event={"ID":"56291e58-e3e7-47a7-844d-3a4b13ba7d00","Type":"ContainerDied","Data":"46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d"} Nov 04 01:47:17.358919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.358714 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97" event={"ID":"56291e58-e3e7-47a7-844d-3a4b13ba7d00","Type":"ContainerDied","Data":"0931f5cc91a336d1fee80327b98fc4c04bc3e14d91a158fa235d0624eb8e16ac"} Nov 04 01:47:17.358919 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.358743 5051 scope.go:122] "RemoveContainer" containerID="46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d" Nov 04 01:47:17.360210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.360184 5051 generic.go:356] "Generic (PLEG): container finished" podID="44ef8d77-d370-4d39-9812-73032fbed1ee" containerID="904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638" exitCode=0 Nov 04 01:47:17.360348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.360229 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" Nov 04 01:47:17.360348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.360274 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" event={"ID":"44ef8d77-d370-4d39-9812-73032fbed1ee","Type":"ContainerDied","Data":"904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638"} Nov 04 01:47:17.360348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.360318 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp" event={"ID":"44ef8d77-d370-4d39-9812-73032fbed1ee","Type":"ContainerDied","Data":"fba5024d2fe1a10ba9346d16dfca01c1f4fbfd23fa4e9580bf990d7287d08e93"} Nov 04 01:47:17.368980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.368955 5051 scope.go:122] "RemoveContainer" containerID="46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d" Nov 04 01:47:17.369225 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:47:17.369205 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d\": container with ID starting with 46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d not found: ID does not exist" containerID="46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d" Nov 04 01:47:17.369301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.369230 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d"} err="failed to get container status \"46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d\": rpc error: code = NotFound desc = could not find container \"46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d\": container with ID starting with 46ce5092d9a48894bf3ca710b42671c52073a27f4beff7d3a45459009639384d not found: ID does not exist" Nov 04 01:47:17.369301 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.369246 5051 scope.go:122] "RemoveContainer" containerID="904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638" Nov 04 01:47:17.377679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.377657 5051 scope.go:122] "RemoveContainer" containerID="904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638" Nov 04 01:47:17.378015 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:47:17.377989 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638\": container with ID starting with 904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638 not found: ID does not exist" containerID="904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638" Nov 04 01:47:17.378171 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.378020 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638"} err="failed to get container status \"904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638\": rpc error: code = NotFound desc = could not find container \"904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638\": container with ID starting with 904414cd9644e8423ee342a642b35608e5f2242ee1732113f1adf8045a986638 not found: ID does not exist" Nov 04 01:47:17.378871 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.378850 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97"] Nov 04 01:47:17.383388 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.383366 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-2ba03d-fdcfddcf6bxt97"] Nov 04 01:47:17.394268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.394246 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp"] Nov 04 01:47:17.398278 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:17.398256 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2ba03d-56fb9h8mcp"] Nov 04 01:47:18.292224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.292169 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:47:18.369625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.369591 5051 generic.go:356] "Generic (PLEG): container finished" podID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerID="393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c" exitCode=0 Nov 04 01:47:18.369771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.369632 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" Nov 04 01:47:18.369771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.369674 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" event={"ID":"7dd36e15-81ce-4e9b-81ab-70444170a621","Type":"ContainerDied","Data":"393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c"} Nov 04 01:47:18.369771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.369718 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs" event={"ID":"7dd36e15-81ce-4e9b-81ab-70444170a621","Type":"ContainerDied","Data":"0a8537b2a10f87774ea20fd79324d5b2b3ef2490036087ab9c78688a08374e95"} Nov 04 01:47:18.369771 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.369740 5051 scope.go:122] "RemoveContainer" containerID="393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c" Nov 04 01:47:18.378926 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.378909 5051 scope.go:122] "RemoveContainer" containerID="7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e" Nov 04 01:47:18.387552 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.387525 5051 scope.go:122] "RemoveContainer" containerID="393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c" Nov 04 01:47:18.387834 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:47:18.387813 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c\": container with ID starting with 393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c not found: ID does not exist" containerID="393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c" Nov 04 01:47:18.387907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.387842 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c"} err="failed to get container status \"393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c\": rpc error: code = NotFound desc = could not find container \"393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c\": container with ID starting with 393f9629ea9462b34a79b000aa11087bc1bc290575b4c689ef6ead089646600c not found: ID does not exist" Nov 04 01:47:18.387907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.387859 5051 scope.go:122] "RemoveContainer" containerID="7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e" Nov 04 01:47:18.388101 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:47:18.388082 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e\": container with ID starting with 7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e not found: ID does not exist" containerID="7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e" Nov 04 01:47:18.388257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.388109 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e"} err="failed to get container status \"7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e\": rpc error: code = NotFound desc = could not find container \"7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e\": container with ID starting with 7890113dcb768fe788ca39e87d2ab12227b29ade112013592ca3dd8ee0089f2e not found: ID does not exist" Nov 04 01:47:18.389364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389347 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dd36e15-81ce-4e9b-81ab-70444170a621-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd36e15-81ce-4e9b-81ab-70444170a621-istiod-ca-cert\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389385 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-podinfo\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389429 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389416 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-data\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389437 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-kube-api-access-56qqc\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-kube-api-access-56qqc\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389476 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-credential-socket\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389515 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-token\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-token\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389593 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-envoy\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389654 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-certs\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389710 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389689 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-socket\") pod \"7dd36e15-81ce-4e9b-81ab-70444170a621\" (UID: \"7dd36e15-81ce-4e9b-81ab-70444170a621\") " Nov 04 01:47:18.389863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389715 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-data" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:18.389863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389751 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-credential-socket" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:18.389863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389784 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd36e15-81ce-4e9b-81ab-70444170a621-istiod-ca-cert" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:47:18.390022 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389961 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-certs" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:18.390022 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.389986 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-socket" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:18.390185 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.390169 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.390257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.390188 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.390257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.390204 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.390257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.390213 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.390257 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.390222 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd36e15-81ce-4e9b-81ab-70444170a621-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.391736 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.391709 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-kube-api-access-56qqc" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "kube-api-access-56qqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:18.391966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.391945 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-envoy" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:47:18.392036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.391982 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-podinfo" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:47:18.392077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.392044 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-token" pod "7dd36e15-81ce-4e9b-81ab-70444170a621" (UID: "7dd36e15-81ce-4e9b-81ab-70444170a621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:47:18.491463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.491440 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.491463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.491461 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56qqc\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-kube-api-access-56qqc\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.491617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.491471 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.491617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.491480 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd36e15-81ce-4e9b-81ab-70444170a621-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:47:18.694359 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.694334 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs"] Nov 04 01:47:18.699515 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:18.699491 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-2ba03d-openshift-default-67b6bf79bfptwjs"] Nov 04 01:47:19.194666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:19.194631 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ef8d77-d370-4d39-9812-73032fbed1ee" path="/var/lib/kubelet/pods/44ef8d77-d370-4d39-9812-73032fbed1ee/volumes" Nov 04 01:47:19.195044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:19.195030 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56291e58-e3e7-47a7-844d-3a4b13ba7d00" path="/var/lib/kubelet/pods/56291e58-e3e7-47a7-844d-3a4b13ba7d00/volumes" Nov 04 01:47:19.195497 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:19.195482 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" path="/var/lib/kubelet/pods/7dd36e15-81ce-4e9b-81ab-70444170a621/volumes" Nov 04 01:47:26.038032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:26.037377 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:47:26.038032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:26.037465 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:47:33.665331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.665295 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv"] Nov 04 01:47:33.666005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.665981 5051 cpu_manager.go:414] "containerMap: removing container" podUID="56291e58-e3e7-47a7-844d-3a4b13ba7d00" containerName="payload-processing" Nov 04 01:47:33.666101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666005 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56291e58-e3e7-47a7-844d-3a4b13ba7d00" containerName="payload-processing" Nov 04 01:47:33.666101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666045 5051 cpu_manager.go:414] "containerMap: removing container" podUID="8e061270-13d8-46b4-bca4-f394723a70ea" containerName="revoke-keys" Nov 04 01:47:33.666101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666054 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e061270-13d8-46b4-bca4-f394723a70ea" containerName="revoke-keys" Nov 04 01:47:33.666101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666079 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" Nov 04 01:47:33.666101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666088 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" Nov 04 01:47:33.666399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666109 5051 cpu_manager.go:414] "containerMap: removing container" podUID="44ef8d77-d370-4d39-9812-73032fbed1ee" containerName="payload-pre-processing" Nov 04 01:47:33.666399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666131 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ef8d77-d370-4d39-9812-73032fbed1ee" containerName="payload-pre-processing" Nov 04 01:47:33.666399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666146 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" Nov 04 01:47:33.666399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666154 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" Nov 04 01:47:33.666399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666262 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ef8d77-d370-4d39-9812-73032fbed1ee" containerName="payload-pre-processing" Nov 04 01:47:33.666399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666279 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" Nov 04 01:47:33.666399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666291 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="56291e58-e3e7-47a7-844d-3a4b13ba7d00" containerName="payload-processing" Nov 04 01:47:33.666399 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.666306 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e061270-13d8-46b4-bca4-f394723a70ea" containerName="revoke-keys" Nov 04 01:47:33.671190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.671166 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.677790 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.677761 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-eab16c40-kserve-self-signed-certs\"" Nov 04 01:47:33.680557 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.680531 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv"] Nov 04 01:47:33.721075 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.721045 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tls-certs\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.721226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.721086 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7tlt\" (UniqueName: \"kubernetes.io/projected/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-kube-api-access-c7tlt\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.721226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.721195 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-dshm\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.721351 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.721244 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-model-cache\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.721419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.721339 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-home\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.721419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.721403 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tmp-dir\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.822652 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.822618 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7tlt\" (UniqueName: \"kubernetes.io/projected/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-kube-api-access-c7tlt\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.822834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.822678 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-dshm\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.822834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.822719 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-model-cache\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.822834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.822754 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-home\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.822834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.822787 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tmp-dir\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.823047 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.822851 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tls-certs\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.823181 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.823157 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-model-cache\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.823267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.823198 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-home\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.823330 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.823285 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tmp-dir\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.825002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.824978 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-dshm\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.825378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.825348 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tls-certs\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.835012 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.834993 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7tlt\" (UniqueName: \"kubernetes.io/projected/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-kube-api-access-c7tlt\") pod \"e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:33.985491 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:33.985460 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:34.107917 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:47:34.107876 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:47:34.264779 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:34.264749 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv"] Nov 04 01:47:34.267470 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:47:34.267434 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd7056d7_0204_451d_bec4_f8c1c0a5ad50.slice/crio-c005128b09a90afaa7dd84c71f5dbeac2f88a39d70dfd3f7c1e059afd465f58b WatchSource:0}: Error finding container c005128b09a90afaa7dd84c71f5dbeac2f88a39d70dfd3f7c1e059afd465f58b: Status 404 returned error can't find the container with id c005128b09a90afaa7dd84c71f5dbeac2f88a39d70dfd3f7c1e059afd465f58b Nov 04 01:47:34.514828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:34.514739 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" event={"ID":"dd7056d7-0204-451d-bec4-f8c1c0a5ad50","Type":"ContainerStarted","Data":"11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f"} Nov 04 01:47:34.514828 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:34.514785 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" event={"ID":"dd7056d7-0204-451d-bec4-f8c1c0a5ad50","Type":"ContainerStarted","Data":"c005128b09a90afaa7dd84c71f5dbeac2f88a39d70dfd3f7c1e059afd465f58b"} Nov 04 01:47:34.985545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:34.985508 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:43.198352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:43.198325 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:47:43.222716 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:43.222693 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:47:43.986336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:43.986299 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:43.998957 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:43.998930 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:47:44.019097 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:44.019054 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" podStartSLOduration=11.019041332 podStartE2EDuration="11.019041332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:47:34.534374755 +0000 UTC m=+2993.506123638" watchObservedRunningTime="2026-11-04 01:47:44.019041332 +0000 UTC m=+3002.990790202" Nov 04 01:47:44.629364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:47:44.629331 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:48:02.388777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:02.388740 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv"] Nov 04 01:48:04.129271 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:48:04.129147 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:48:11.008240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:11.008198 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:48:11.008890 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:11.008853 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:48:17.424143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.420651 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" podUID="dd7056d7-0204-451d-bec4-f8c1c0a5ad50" containerName="main" containerID="cri-o://11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f" gracePeriod=45 Nov 04 01:48:17.623206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.619415 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" podUID="dd7056d7-0204-451d-bec4-f8c1c0a5ad50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": dial tcp 10.132.0.243:8000: connect: connection refused" Nov 04 01:48:17.852789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.852749 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:48:17.942310 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.942267 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-model-cache\") pod \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " Nov 04 01:48:17.942635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.942601 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-model-cache" pod "dd7056d7-0204-451d-bec4-f8c1c0a5ad50" (UID: "dd7056d7-0204-451d-bec4-f8c1c0a5ad50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:17.942768 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.942623 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tls-certs\") pod \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " Nov 04 01:48:17.942973 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.942944 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-dshm\") pod \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " Nov 04 01:48:17.943541 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.943519 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tmp-dir\") pod \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " Nov 04 01:48:17.944103 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.944082 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-kube-api-access-c7tlt\" (UniqueName: \"kubernetes.io/projected/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-kube-api-access-c7tlt\") pod \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " Nov 04 01:48:17.944638 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.943957 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tmp-dir" pod "dd7056d7-0204-451d-bec4-f8c1c0a5ad50" (UID: "dd7056d7-0204-451d-bec4-f8c1c0a5ad50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:17.946277 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.946231 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-home" pod "dd7056d7-0204-451d-bec4-f8c1c0a5ad50" (UID: "dd7056d7-0204-451d-bec4-f8c1c0a5ad50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:17.946405 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.944588 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-home\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-home\") pod \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\" (UID: \"dd7056d7-0204-451d-bec4-f8c1c0a5ad50\") " Nov 04 01:48:17.947102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.947066 5051 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tmp-dir\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.947265 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.947249 5051 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-home\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.947371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.947357 5051 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-model-cache\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:17.948573 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.948548 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tls-certs" pod "dd7056d7-0204-451d-bec4-f8c1c0a5ad50" (UID: "dd7056d7-0204-451d-bec4-f8c1c0a5ad50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:48:17.950190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.950086 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-kube-api-access-c7tlt" pod "dd7056d7-0204-451d-bec4-f8c1c0a5ad50" (UID: "dd7056d7-0204-451d-bec4-f8c1c0a5ad50"). InnerVolumeSpecName "kube-api-access-c7tlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:48:17.950654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:17.950616 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-dshm" pod "dd7056d7-0204-451d-bec4-f8c1c0a5ad50" (UID: "dd7056d7-0204-451d-bec4-f8c1c0a5ad50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:48:18.049091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.049048 5051 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-tls-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:18.049091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.049092 5051 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-dshm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:18.049091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.049108 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7tlt\" (UniqueName: \"kubernetes.io/projected/dd7056d7-0204-451d-bec4-f8c1c0a5ad50-kube-api-access-c7tlt\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:48:18.126799 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.126753 5051 generic.go:356] "Generic (PLEG): container finished" podID="dd7056d7-0204-451d-bec4-f8c1c0a5ad50" containerID="11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f" exitCode=0 Nov 04 01:48:18.127026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.126822 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" Nov 04 01:48:18.127026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.126857 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" event={"ID":"dd7056d7-0204-451d-bec4-f8c1c0a5ad50","Type":"ContainerDied","Data":"11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f"} Nov 04 01:48:18.127026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.126912 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv" event={"ID":"dd7056d7-0204-451d-bec4-f8c1c0a5ad50","Type":"ContainerDied","Data":"c005128b09a90afaa7dd84c71f5dbeac2f88a39d70dfd3f7c1e059afd465f58b"} Nov 04 01:48:18.127026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.126938 5051 scope.go:122] "RemoveContainer" containerID="11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f" Nov 04 01:48:18.145160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.143106 5051 scope.go:122] "RemoveContainer" containerID="11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f" Nov 04 01:48:18.145160 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:48:18.143463 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f\": container with ID starting with 11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f not found: ID does not exist" containerID="11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f" Nov 04 01:48:18.145160 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.143505 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f"} err="failed to get container status \"11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f\": rpc error: code = NotFound desc = could not find container \"11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f\": container with ID starting with 11065e116a883ded11fd2e5398b309180db6c7c6abf5324cf97e2dfcfac2026f not found: ID does not exist" Nov 04 01:48:18.156945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.156917 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv"] Nov 04 01:48:18.160407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:18.160371 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-eab16c40-kserve-7cbdc866f-442xv"] Nov 04 01:48:19.206143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:19.202601 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7056d7-0204-451d-bec4-f8c1c0a5ad50" path="/var/lib/kubelet/pods/dd7056d7-0204-451d-bec4-f8c1c0a5ad50/volumes" Nov 04 01:48:34.105955 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:48:34.105916 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:48:42.118415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.118373 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv"] Nov 04 01:48:42.118978 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.118957 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dd7056d7-0204-451d-bec4-f8c1c0a5ad50" containerName="main" Nov 04 01:48:42.119071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.118978 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7056d7-0204-451d-bec4-f8c1c0a5ad50" containerName="main" Nov 04 01:48:42.119071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.119056 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7056d7-0204-451d-bec4-f8c1c0a5ad50" containerName="main" Nov 04 01:48:42.119071 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.119064 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dd36e15-81ce-4e9b-81ab-70444170a621" containerName="istio-proxy" Nov 04 01:48:42.122303 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.122278 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.132850 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.132812 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b1e8de2d-openshift-default-dockercfg-jb8bg\"" Nov 04 01:48:42.144232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.144201 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv"] Nov 04 01:48:42.207836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.207803 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-credential-socket\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.208069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.207843 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-certs\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.208069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.207873 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-data\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.208069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.208047 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-envoy\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.208271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.208096 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-podinfo\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.208271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.208177 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-token\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.208271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.208205 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istiod-ca-cert\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.208271 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.208225 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-socket\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.208425 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.208308 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qf4g\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-kube-api-access-8qf4g\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.309907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.309835 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istiod-ca-cert\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.309907 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.309889 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-socket\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.309939 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qf4g\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-kube-api-access-8qf4g\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310001 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-credential-socket\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310031 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-certs\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310065 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-data\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310163 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-envoy\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310195 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-podinfo\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310249 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310227 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-token\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310382 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-socket\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310442 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-credential-socket\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310633 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-data\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310747 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-certs\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.310847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.310758 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istiod-ca-cert\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.312873 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.312840 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-envoy\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.313152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.313136 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-podinfo\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.325864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.325833 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-token\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.326006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.325884 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qf4g\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-kube-api-access-8qf4g\") pod \"e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.436360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.436264 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:42.732752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.732726 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv"] Nov 04 01:48:42.734342 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:48:42.734315 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde6cbbe_11de_4f98_b0bd_8379be9022cf.slice/crio-1a0b8af0081490ef649fdfbb3ca41d90ffe7328add4c21375b5413c7ca77a5e4 WatchSource:0}: Error finding container 1a0b8af0081490ef649fdfbb3ca41d90ffe7328add4c21375b5413c7ca77a5e4: Status 404 returned error can't find the container with id 1a0b8af0081490ef649fdfbb3ca41d90ffe7328add4c21375b5413c7ca77a5e4 Nov 04 01:48:42.736721 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.736683 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:48:42.736860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.736770 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:48:42.736860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:42.736819 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:48:43.301263 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:43.301229 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" event={"ID":"dde6cbbe-11de-4f98-b0bd-8379be9022cf","Type":"ContainerStarted","Data":"927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3"} Nov 04 01:48:43.301732 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:43.301285 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" event={"ID":"dde6cbbe-11de-4f98-b0bd-8379be9022cf","Type":"ContainerStarted","Data":"1a0b8af0081490ef649fdfbb3ca41d90ffe7328add4c21375b5413c7ca77a5e4"} Nov 04 01:48:43.331505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:43.331441 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" podStartSLOduration=1.331421833 podStartE2EDuration="1.331421833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:43.329289508 +0000 UTC m=+3062.301038379" watchObservedRunningTime="2026-11-04 01:48:43.331421833 +0000 UTC m=+3062.303170705" Nov 04 01:48:43.436619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:43.436581 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:43.441993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:43.441966 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:44.307538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:44.307505 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:44.308673 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:44.308648 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:48:46.173867 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.173831 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j"] Nov 04 01:48:46.177679 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.177661 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:46.181788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.181770 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b1e8de2d-dockercfg-9lzvm\"" Nov 04 01:48:46.182995 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.182974 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b1e8de2d\"" Nov 04 01:48:46.194986 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.194963 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j"] Nov 04 01:48:46.203546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.203519 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6"] Nov 04 01:48:46.207293 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.207274 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:46.219024 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.218981 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6"] Nov 04 01:48:46.253746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.253713 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfp5q\" (UniqueName: \"kubernetes.io/projected/3d2d7670-f17e-4f89-b306-4607b8b9c12f-kube-api-access-lfp5q\") pod \"payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6\" (UID: \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:46.253938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.253841 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvq6\" (UniqueName: \"kubernetes.io/projected/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-kube-api-access-pmvq6\") pod \"payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j\" (UID: \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:46.253938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.253868 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j\" (UID: \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:46.254253 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.254233 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2d7670-f17e-4f89-b306-4607b8b9c12f-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6\" (UID: \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:46.355565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.355527 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2d7670-f17e-4f89-b306-4607b8b9c12f-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6\" (UID: \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:46.355770 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.355689 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfp5q\" (UniqueName: \"kubernetes.io/projected/3d2d7670-f17e-4f89-b306-4607b8b9c12f-kube-api-access-lfp5q\") pod \"payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6\" (UID: \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:46.355865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.355767 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvq6\" (UniqueName: \"kubernetes.io/projected/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-kube-api-access-pmvq6\") pod \"payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j\" (UID: \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:46.355865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.355801 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j\" (UID: \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:46.356427 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.356405 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2d7670-f17e-4f89-b306-4607b8b9c12f-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6\" (UID: \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:46.356556 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.356537 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j\" (UID: \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:46.364620 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.364593 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvq6\" (UniqueName: \"kubernetes.io/projected/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-kube-api-access-pmvq6\") pod \"payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j\" (UID: \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:46.364802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.364779 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfp5q\" (UniqueName: \"kubernetes.io/projected/3d2d7670-f17e-4f89-b306-4607b8b9c12f-kube-api-access-lfp5q\") pod \"payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6\" (UID: \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:46.491888 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.491857 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:46.527973 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.527944 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:46.790028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.788453 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j"] Nov 04 01:48:46.827412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:46.827337 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6"] Nov 04 01:48:46.830504 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:48:46.830472 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2d7670_f17e_4f89_b306_4607b8b9c12f.slice/crio-403bd325386d3e2f110f861fe7209cbf12b3804747efff30943715bf1c5d2116 WatchSource:0}: Error finding container 403bd325386d3e2f110f861fe7209cbf12b3804747efff30943715bf1c5d2116: Status 404 returned error can't find the container with id 403bd325386d3e2f110f861fe7209cbf12b3804747efff30943715bf1c5d2116 Nov 04 01:48:47.324739 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:47.324687 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" event={"ID":"3d2d7670-f17e-4f89-b306-4607b8b9c12f","Type":"ContainerStarted","Data":"ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b"} Nov 04 01:48:47.325217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:47.324762 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:47.325217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:47.324788 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" event={"ID":"3d2d7670-f17e-4f89-b306-4607b8b9c12f","Type":"ContainerStarted","Data":"403bd325386d3e2f110f861fe7209cbf12b3804747efff30943715bf1c5d2116"} Nov 04 01:48:47.326389 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:47.326359 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" event={"ID":"12bb1af8-d600-43d0-ad1d-812b7d5ac3de","Type":"ContainerStarted","Data":"1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d"} Nov 04 01:48:47.326545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:47.326409 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" event={"ID":"12bb1af8-d600-43d0-ad1d-812b7d5ac3de","Type":"ContainerStarted","Data":"ea6131229552dc2f5caf897a9430065d3479354d6b3748ccf5e508abca8036dd"} Nov 04 01:48:47.326545 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:47.326435 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:47.372560 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:47.372512 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" podStartSLOduration=1.37249873 podStartE2EDuration="1.37249873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:47.370678913 +0000 UTC m=+3066.342427785" watchObservedRunningTime="2026-11-04 01:48:47.37249873 +0000 UTC m=+3066.344247619" Nov 04 01:48:47.408072 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:47.408027 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" podStartSLOduration=1.408014282 podStartE2EDuration="1.408014282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:48:47.407912615 +0000 UTC m=+3066.379661483" watchObservedRunningTime="2026-11-04 01:48:47.408014282 +0000 UTC m=+3066.379763152" Nov 04 01:48:50.135534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:50.135500 5051 scope.go:122] "RemoveContainer" containerID="66ca42e751c4728a61416fb9dd65148f3efa8c8036df8be4d7c856a473c23768" Nov 04 01:48:58.334868 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:58.334836 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:48:58.335382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:58.335283 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:48:59.308899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:59.308861 5051 patch_prober.go:29] interesting pod/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 04 01:48:59.309152 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:59.308919 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 04 01:48:59.417142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:59.417087 5051 generic.go:356] "Generic (PLEG): container finished" podID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerID="927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3" exitCode=0 Nov 04 01:48:59.417521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:59.417163 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" event={"ID":"dde6cbbe-11de-4f98-b0bd-8379be9022cf","Type":"ContainerDied","Data":"927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3"} Nov 04 01:48:59.417592 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:59.417574 5051 scope.go:122] "RemoveContainer" containerID="927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3" Nov 04 01:48:59.418711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:59.418681 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:48:59.418772 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:59.418752 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:48:59.418809 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:48:59.418797 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:49:00.425814 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.425780 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" event={"ID":"dde6cbbe-11de-4f98-b0bd-8379be9022cf","Type":"ContainerStarted","Data":"59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905"} Nov 04 01:49:00.436636 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.436612 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:49:00.436826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.436804 5051 patch_prober.go:29] interesting pod/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 04 01:49:00.436874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.436849 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 04 01:49:00.626166 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.626104 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr"] Nov 04 01:49:00.630347 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.630320 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.635654 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.635623 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:49:00.635777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.635736 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:49:00.635777 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.635770 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:49:00.636069 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.636050 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpjkv\"" Nov 04 01:49:00.638962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.638938 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr"] Nov 04 01:49:00.734525 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.734491 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkw6\" (UniqueName: \"kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.734702 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.734539 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.734749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.734717 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.835450 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.835420 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.835622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.835577 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkw6\" (UniqueName: \"kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.835622 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.835611 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.836360 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.836334 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.837938 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:00.837915 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:00.841593 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:49:00.841569 5051 projected.go:196] Error preparing data for projected volume kube-api-access-jpkw6 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 04 01:49:00.841707 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:49:00.841640 5051 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6 podName:333d4b83-0be8-46dd-907d-76e03aefd2d5 nodeName:}" failed. No retries permitted until 2026-11-04 01:49:01.341617889 +0000 UTC m=+3080.313366737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jpkw6" (UniqueName: "kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6") pod "maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" (UID: "333d4b83-0be8-46dd-907d-76e03aefd2d5") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 04 01:49:01.436616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:01.436569 5051 patch_prober.go:29] interesting pod/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 04 01:49:01.437038 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:01.436639 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 04 01:49:01.442163 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:01.442111 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkw6\" (UniqueName: \"kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:01.468142 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:01.467842 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkw6\" (UniqueName: \"kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6\") pod \"maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:01.545018 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:01.544993 5051 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" secret="" err="secret \"maas-api-cleanup-dockercfg-mpjkv\" not found" Nov 04 01:49:01.545168 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:01.545035 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:01.843874 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:01.843826 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr"] Nov 04 01:49:01.846316 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:49:01.846279 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333d4b83_0be8_46dd_907d_76e03aefd2d5.slice/crio-81494d10c3664e2281f332f5fc0302827d1a5882527ac068519b8d29c7d1155f WatchSource:0}: Error finding container 81494d10c3664e2281f332f5fc0302827d1a5882527ac068519b8d29c7d1155f: Status 404 returned error can't find the container with id 81494d10c3664e2281f332f5fc0302827d1a5882527ac068519b8d29c7d1155f Nov 04 01:49:02.438294 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:02.438212 5051 patch_prober.go:29] interesting pod/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 04 01:49:02.438294 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:02.438270 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 04 01:49:02.441023 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:02.440992 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" event={"ID":"333d4b83-0be8-46dd-907d-76e03aefd2d5","Type":"ContainerStarted","Data":"804b36e8a006c1d032094e5d373cfc48ce5f0898f4ebdd31e64f1e781f7bcc5d"} Nov 04 01:49:02.441208 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:02.441047 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" event={"ID":"333d4b83-0be8-46dd-907d-76e03aefd2d5","Type":"ContainerStarted","Data":"81494d10c3664e2281f332f5fc0302827d1a5882527ac068519b8d29c7d1155f"} Nov 04 01:49:02.441287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:02.441268 5051 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" secret="" err="secret \"maas-api-cleanup-dockercfg-mpjkv\" not found" Nov 04 01:49:02.473145 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:02.468138 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" podStartSLOduration=2.4681038490000002 podStartE2EDuration="2.468103849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:49:02.465732807 +0000 UTC m=+3081.437481681" watchObservedRunningTime="2026-11-04 01:49:02.468103849 +0000 UTC m=+3081.439852721" Nov 04 01:49:03.437454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:03.437416 5051 patch_prober.go:29] interesting pod/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 04 01:49:03.437646 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:03.437475 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 04 01:49:03.448254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:03.448224 5051 generic.go:356] "Generic (PLEG): container finished" podID="333d4b83-0be8-46dd-907d-76e03aefd2d5" containerID="804b36e8a006c1d032094e5d373cfc48ce5f0898f4ebdd31e64f1e781f7bcc5d" exitCode=0 Nov 04 01:49:03.448632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:03.448315 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" event={"ID":"333d4b83-0be8-46dd-907d-76e03aefd2d5","Type":"ContainerDied","Data":"804b36e8a006c1d032094e5d373cfc48ce5f0898f4ebdd31e64f1e781f7bcc5d"} Nov 04 01:49:04.106072 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:49:04.106033 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:49:04.436826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.436736 5051 patch_prober.go:29] interesting pod/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 04 01:49:04.436826 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.436791 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 04 01:49:04.638309 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.638287 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:04.802997 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.802963 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6\" (UniqueName: \"kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6\") pod \"333d4b83-0be8-46dd-907d-76e03aefd2d5\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " Nov 04 01:49:04.803201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.803015 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-xks-ca\") pod \"333d4b83-0be8-46dd-907d-76e03aefd2d5\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " Nov 04 01:49:04.803201 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.803060 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-openshift-ca\") pod \"333d4b83-0be8-46dd-907d-76e03aefd2d5\" (UID: \"333d4b83-0be8-46dd-907d-76e03aefd2d5\") " Nov 04 01:49:04.803925 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.803876 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-openshift-ca" pod "333d4b83-0be8-46dd-907d-76e03aefd2d5" (UID: "333d4b83-0be8-46dd-907d-76e03aefd2d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:04.805509 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.805486 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-xks-ca" pod "333d4b83-0be8-46dd-907d-76e03aefd2d5" (UID: "333d4b83-0be8-46dd-907d-76e03aefd2d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:49:04.806756 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.806736 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6" pod "333d4b83-0be8-46dd-907d-76e03aefd2d5" (UID: "333d4b83-0be8-46dd-907d-76e03aefd2d5"). InnerVolumeSpecName "kube-api-access-jpkw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:04.904403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.904364 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:04.904403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.904396 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333d4b83-0be8-46dd-907d-76e03aefd2d5-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:04.904403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:04.904408 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkw6\" (UniqueName: \"kubernetes.io/projected/333d4b83-0be8-46dd-907d-76e03aefd2d5-kube-api-access-jpkw6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:05.440440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:05.440408 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:49:05.440694 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:05.440674 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:49:05.441447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:05.441426 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:49:05.470005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:05.469919 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" event={"ID":"333d4b83-0be8-46dd-907d-76e03aefd2d5","Type":"ContainerDied","Data":"81494d10c3664e2281f332f5fc0302827d1a5882527ac068519b8d29c7d1155f"} Nov 04 01:49:05.470005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:05.469978 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81494d10c3664e2281f332f5fc0302827d1a5882527ac068519b8d29c7d1155f" Nov 04 01:49:05.470005 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:05.469939 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr" Nov 04 01:49:10.736245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:10.736208 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j"] Nov 04 01:49:10.737193 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:10.737162 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6"] Nov 04 01:49:11.203864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:11.203773 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr"] Nov 04 01:49:11.217948 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:11.217904 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1e8de2d-6t8cr"] Nov 04 01:49:11.892151 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:11.892102 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv"] Nov 04 01:49:11.892651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:11.892408 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" containerID="cri-o://59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905" gracePeriod=30 Nov 04 01:49:13.187091 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.187056 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333d4b83-0be8-46dd-907d-76e03aefd2d5" path="/var/lib/kubelet/pods/333d4b83-0be8-46dd-907d-76e03aefd2d5/volumes" Nov 04 01:49:13.634002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.633966 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb"] Nov 04 01:49:13.634615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.634588 5051 cpu_manager.go:414] "containerMap: removing container" podUID="333d4b83-0be8-46dd-907d-76e03aefd2d5" containerName="revoke-keys" Nov 04 01:49:13.634615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.634612 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333d4b83-0be8-46dd-907d-76e03aefd2d5" containerName="revoke-keys" Nov 04 01:49:13.634789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.634735 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="333d4b83-0be8-46dd-907d-76e03aefd2d5" containerName="revoke-keys" Nov 04 01:49:13.638276 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.638250 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.645908 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.645522 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-442e8c-openshift-default-dockercfg-66w7m\"" Nov 04 01:49:13.658954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.658921 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb"] Nov 04 01:49:13.689214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689115 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-data\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.689379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689238 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-envoy\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.689379 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689327 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1304976-517f-4a62-99cd-9404669e2d6a-istio-podinfo\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.689501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689416 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-socket\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.689501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689460 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1304976-517f-4a62-99cd-9404669e2d6a-istiod-ca-cert\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.689623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689501 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-istio-token\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.689623 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689575 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8zrm\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-kube-api-access-g8zrm\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.689733 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689619 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-certs\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.689733 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.689654 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-credential-socket\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790248 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-socket\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790311 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1304976-517f-4a62-99cd-9404669e2d6a-istiod-ca-cert\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790351 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-istio-token\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790416 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8zrm\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-kube-api-access-g8zrm\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790462 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-certs\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790492 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-credential-socket\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790516 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-data\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790542 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-envoy\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790588 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1304976-517f-4a62-99cd-9404669e2d6a-istio-podinfo\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790755 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-socket\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790861 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-credential-socket\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.790945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.790933 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-certs\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.791086 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.791037 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-data\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.791460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.791435 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1304976-517f-4a62-99cd-9404669e2d6a-istiod-ca-cert\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.794068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.794043 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1304976-517f-4a62-99cd-9404669e2d6a-istio-podinfo\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.794259 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.794224 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-envoy\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.802737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.802715 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8zrm\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-kube-api-access-g8zrm\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.804076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.804044 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-istio-token\") pod \"e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:13.965068 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:13.964981 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:14.260077 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:14.260044 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb"] Nov 04 01:49:14.261534 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:49:14.261506 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1304976_517f_4a62_99cd_9404669e2d6a.slice/crio-7825d8635abcd4b85e3416d9207eec642063724e86dbcb89460dc983671abc46 WatchSource:0}: Error finding container 7825d8635abcd4b85e3416d9207eec642063724e86dbcb89460dc983671abc46: Status 404 returned error can't find the container with id 7825d8635abcd4b85e3416d9207eec642063724e86dbcb89460dc983671abc46 Nov 04 01:49:14.263786 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:14.263740 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:49:14.263860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:14.263819 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:49:14.263860 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:14.263854 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:49:14.520722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:14.520633 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" event={"ID":"b1304976-517f-4a62-99cd-9404669e2d6a","Type":"ContainerStarted","Data":"2b291820cd110798dce2f81ee1593dab15ccb3eeeb7aed2ac99b53334e38a29a"} Nov 04 01:49:14.520722 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:14.520701 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" event={"ID":"b1304976-517f-4a62-99cd-9404669e2d6a","Type":"ContainerStarted","Data":"7825d8635abcd4b85e3416d9207eec642063724e86dbcb89460dc983671abc46"} Nov 04 01:49:14.544788 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:14.544731 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" podStartSLOduration=1.544717404 podStartE2EDuration="1.544717404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:49:14.542695701 +0000 UTC m=+3093.514444555" watchObservedRunningTime="2026-11-04 01:49:14.544717404 +0000 UTC m=+3093.516466274" Nov 04 01:49:14.965146 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:14.965054 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:15.736964 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:15.736887 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" podUID="12bb1af8-d600-43d0-ad1d-812b7d5ac3de" containerName="payload-pre-processing" containerID="cri-o://1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d" gracePeriod=25 Nov 04 01:49:15.738444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:15.738412 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" podUID="3d2d7670-f17e-4f89-b306-4607b8b9c12f" containerName="payload-processing" containerID="cri-o://ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b" gracePeriod=25 Nov 04 01:49:15.975961 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:15.975918 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:16.046013 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.045990 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:49:16.049284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.049263 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:49:16.114674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.114646 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-plugins-config-volume\") pod \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\" (UID: \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\") " Nov 04 01:49:16.114847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.114692 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2d7670-f17e-4f89-b306-4607b8b9c12f-kube-api-access-lfp5q\" (UniqueName: \"kubernetes.io/projected/3d2d7670-f17e-4f89-b306-4607b8b9c12f-kube-api-access-lfp5q\") pod \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\" (UID: \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\") " Nov 04 01:49:16.114847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.114759 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2d7670-f17e-4f89-b306-4607b8b9c12f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2d7670-f17e-4f89-b306-4607b8b9c12f-plugins-config-volume\") pod \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\" (UID: \"3d2d7670-f17e-4f89-b306-4607b8b9c12f\") " Nov 04 01:49:16.114847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.114818 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-kube-api-access-pmvq6\" (UniqueName: \"kubernetes.io/projected/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-kube-api-access-pmvq6\") pod \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\" (UID: \"12bb1af8-d600-43d0-ad1d-812b7d5ac3de\") " Nov 04 01:49:16.115169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.115103 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-plugins-config-volume" pod "12bb1af8-d600-43d0-ad1d-812b7d5ac3de" (UID: "12bb1af8-d600-43d0-ad1d-812b7d5ac3de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:16.115289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.115245 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:16.115289 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.115251 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2d7670-f17e-4f89-b306-4607b8b9c12f-plugins-config-volume" pod "3d2d7670-f17e-4f89-b306-4607b8b9c12f" (UID: "3d2d7670-f17e-4f89-b306-4607b8b9c12f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:16.116990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.116965 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2d7670-f17e-4f89-b306-4607b8b9c12f-kube-api-access-lfp5q" pod "3d2d7670-f17e-4f89-b306-4607b8b9c12f" (UID: "3d2d7670-f17e-4f89-b306-4607b8b9c12f"). InnerVolumeSpecName "kube-api-access-lfp5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:16.117094 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.117015 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-kube-api-access-pmvq6" pod "12bb1af8-d600-43d0-ad1d-812b7d5ac3de" (UID: "12bb1af8-d600-43d0-ad1d-812b7d5ac3de"). InnerVolumeSpecName "kube-api-access-pmvq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:16.215786 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.215745 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfp5q\" (UniqueName: \"kubernetes.io/projected/3d2d7670-f17e-4f89-b306-4607b8b9c12f-kube-api-access-lfp5q\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:16.215786 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.215781 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d2d7670-f17e-4f89-b306-4607b8b9c12f-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:16.215786 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.215791 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvq6\" (UniqueName: \"kubernetes.io/projected/12bb1af8-d600-43d0-ad1d-812b7d5ac3de-kube-api-access-pmvq6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:16.534621 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.534581 5051 generic.go:356] "Generic (PLEG): container finished" podID="3d2d7670-f17e-4f89-b306-4607b8b9c12f" containerID="ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b" exitCode=0 Nov 04 01:49:16.534827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.534635 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" Nov 04 01:49:16.534827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.534665 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" event={"ID":"3d2d7670-f17e-4f89-b306-4607b8b9c12f","Type":"ContainerDied","Data":"ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b"} Nov 04 01:49:16.534827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.534719 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6" event={"ID":"3d2d7670-f17e-4f89-b306-4607b8b9c12f","Type":"ContainerDied","Data":"403bd325386d3e2f110f861fe7209cbf12b3804747efff30943715bf1c5d2116"} Nov 04 01:49:16.534827 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.534748 5051 scope.go:122] "RemoveContainer" containerID="ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b" Nov 04 01:49:16.536210 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.536173 5051 generic.go:356] "Generic (PLEG): container finished" podID="12bb1af8-d600-43d0-ad1d-812b7d5ac3de" containerID="1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d" exitCode=0 Nov 04 01:49:16.536331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.536221 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" Nov 04 01:49:16.536331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.536254 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" event={"ID":"12bb1af8-d600-43d0-ad1d-812b7d5ac3de","Type":"ContainerDied","Data":"1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d"} Nov 04 01:49:16.536331 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.536314 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j" event={"ID":"12bb1af8-d600-43d0-ad1d-812b7d5ac3de","Type":"ContainerDied","Data":"ea6131229552dc2f5caf897a9430065d3479354d6b3748ccf5e508abca8036dd"} Nov 04 01:49:16.536797 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.536776 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:16.539205 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.539187 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:16.545327 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.545288 5051 scope.go:122] "RemoveContainer" containerID="ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b" Nov 04 01:49:16.545648 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:49:16.545624 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b\": container with ID starting with ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b not found: ID does not exist" containerID="ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b" Nov 04 01:49:16.545763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.545653 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b"} err="failed to get container status \"ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b\": rpc error: code = NotFound desc = could not find container \"ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b\": container with ID starting with ccad689ee4aa233e72edcfad1a1a86c2cc2eb9f24bcfff4eb90927af80c6083b not found: ID does not exist" Nov 04 01:49:16.545763 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.545680 5051 scope.go:122] "RemoveContainer" containerID="1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d" Nov 04 01:49:16.557440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.556906 5051 scope.go:122] "RemoveContainer" containerID="1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d" Nov 04 01:49:16.557440 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:49:16.557254 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d\": container with ID starting with 1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d not found: ID does not exist" containerID="1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d" Nov 04 01:49:16.557440 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.557285 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d"} err="failed to get container status \"1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d\": rpc error: code = NotFound desc = could not find container \"1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d\": container with ID starting with 1e2786090e5ff6617379868e210d5dfbc41f7d298135aefdb9a1b41915048e7d not found: ID does not exist" Nov 04 01:49:16.580349 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.580311 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6"] Nov 04 01:49:16.584439 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.584384 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1e8de2d-65bc4b6b4-69hg6"] Nov 04 01:49:16.596896 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.596858 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j"] Nov 04 01:49:16.599190 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:16.599161 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1e8de2d-76b9dc94c-bfp7j"] Nov 04 01:49:17.144225 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.144195 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:49:17.189291 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.189220 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12bb1af8-d600-43d0-ad1d-812b7d5ac3de" path="/var/lib/kubelet/pods/12bb1af8-d600-43d0-ad1d-812b7d5ac3de/volumes" Nov 04 01:49:17.189653 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.189639 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2d7670-f17e-4f89-b306-4607b8b9c12f" path="/var/lib/kubelet/pods/3d2d7670-f17e-4f89-b306-4607b8b9c12f/volumes" Nov 04 01:49:17.226042 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226011 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-podinfo\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226098 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-credential-socket\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226141 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-data\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226172 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-token\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-token\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226198 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istiod-ca-cert\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226245 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226231 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-kube-api-access-8qf4g\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-kube-api-access-8qf4g\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226270 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-certs\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226332 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-envoy\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226416 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-data" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:17.226527 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226436 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-socket\") pod \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\" (UID: \"dde6cbbe-11de-4f98-b0bd-8379be9022cf\") " Nov 04 01:49:17.226735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226431 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-credential-socket" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:17.226735 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226659 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-socket" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:17.226853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.226816 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-certs" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:17.227044 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.227022 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.227198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.227050 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.227198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.227068 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.227198 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.227106 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istiod-ca-cert" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:49:17.228598 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.228568 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-podinfo" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:49:17.228705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.228615 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-envoy" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:49:17.228705 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.228634 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-token" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:17.229026 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.229009 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-kube-api-access-8qf4g" pod "dde6cbbe-11de-4f98-b0bd-8379be9022cf" (UID: "dde6cbbe-11de-4f98-b0bd-8379be9022cf"). InnerVolumeSpecName "kube-api-access-8qf4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:49:17.328460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.328423 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.328460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.328454 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.328460 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.328464 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.328717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.328473 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dde6cbbe-11de-4f98-b0bd-8379be9022cf-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.328717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.328483 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qf4g\" (UniqueName: \"kubernetes.io/projected/dde6cbbe-11de-4f98-b0bd-8379be9022cf-kube-api-access-8qf4g\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.328717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.328493 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dde6cbbe-11de-4f98-b0bd-8379be9022cf-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:49:17.543980 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.543941 5051 generic.go:356] "Generic (PLEG): container finished" podID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerID="59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905" exitCode=0 Nov 04 01:49:17.544206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.544003 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" Nov 04 01:49:17.544206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.544021 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" event={"ID":"dde6cbbe-11de-4f98-b0bd-8379be9022cf","Type":"ContainerDied","Data":"59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905"} Nov 04 01:49:17.544206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.544078 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv" event={"ID":"dde6cbbe-11de-4f98-b0bd-8379be9022cf","Type":"ContainerDied","Data":"1a0b8af0081490ef649fdfbb3ca41d90ffe7328add4c21375b5413c7ca77a5e4"} Nov 04 01:49:17.544206 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.544107 5051 scope.go:122] "RemoveContainer" containerID="59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905" Nov 04 01:49:17.554725 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.554710 5051 scope.go:122] "RemoveContainer" containerID="927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3" Nov 04 01:49:17.563486 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.563468 5051 scope.go:122] "RemoveContainer" containerID="59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905" Nov 04 01:49:17.563760 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:49:17.563732 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905\": container with ID starting with 59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905 not found: ID does not exist" containerID="59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905" Nov 04 01:49:17.563845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.563772 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905"} err="failed to get container status \"59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905\": rpc error: code = NotFound desc = could not find container \"59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905\": container with ID starting with 59a163fba23bbabab8e86be4d7e0c4c9e237ec7ed2852a1811808c4a75042905 not found: ID does not exist" Nov 04 01:49:17.563845 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.563795 5051 scope.go:122] "RemoveContainer" containerID="927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3" Nov 04 01:49:17.564066 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:49:17.564048 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3\": container with ID starting with 927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3 not found: ID does not exist" containerID="927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3" Nov 04 01:49:17.564106 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.564073 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3"} err="failed to get container status \"927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3\": rpc error: code = NotFound desc = could not find container \"927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3\": container with ID starting with 927120b926fba2cdb0aec11cdbf2b25b419d13aaae98f0195ac1be4326e2d8c3 not found: ID does not exist" Nov 04 01:49:17.569942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.569918 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv"] Nov 04 01:49:17.573474 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:17.573455 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b1e8de2d-openshift-default-7c49c75775-w48fv"] Nov 04 01:49:18.055717 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.055684 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp"] Nov 04 01:49:18.056348 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056332 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" Nov 04 01:49:18.056403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056351 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" Nov 04 01:49:18.056403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056368 5051 cpu_manager.go:414] "containerMap: removing container" podUID="3d2d7670-f17e-4f89-b306-4607b8b9c12f" containerName="payload-processing" Nov 04 01:49:18.056403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056375 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2d7670-f17e-4f89-b306-4607b8b9c12f" containerName="payload-processing" Nov 04 01:49:18.056505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056426 5051 cpu_manager.go:414] "containerMap: removing container" podUID="12bb1af8-d600-43d0-ad1d-812b7d5ac3de" containerName="payload-pre-processing" Nov 04 01:49:18.056505 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056435 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bb1af8-d600-43d0-ad1d-812b7d5ac3de" containerName="payload-pre-processing" Nov 04 01:49:18.056565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056541 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2d7670-f17e-4f89-b306-4607b8b9c12f" containerName="payload-processing" Nov 04 01:49:18.056565 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056562 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" Nov 04 01:49:18.056629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056570 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bb1af8-d600-43d0-ad1d-812b7d5ac3de" containerName="payload-pre-processing" Nov 04 01:49:18.056629 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.056582 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" Nov 04 01:49:18.061535 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.061510 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:18.069355 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.069325 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp"] Nov 04 01:49:18.073308 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.073277 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-442e8c\"" Nov 04 01:49:18.073708 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.073669 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-442e8c-dockercfg-8mqfc\"" Nov 04 01:49:18.117398 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.117366 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c"] Nov 04 01:49:18.117854 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.117839 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" Nov 04 01:49:18.117904 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.117855 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" containerName="istio-proxy" Nov 04 01:49:18.121132 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.121102 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:18.136113 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.136078 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c"] Nov 04 01:49:18.136539 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.136521 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvt47\" (UniqueName: \"kubernetes.io/projected/6ee97aa7-7dcd-493c-8431-683e30c4055c-kube-api-access-kvt47\") pod \"payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp\" (UID: \"6ee97aa7-7dcd-493c-8431-683e30c4055c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:18.136627 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.136568 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcwk\" (UniqueName: \"kubernetes.io/projected/9ace9183-d35f-431e-bb1b-d09f985d688d-kube-api-access-vjcwk\") pod \"payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c\" (UID: \"9ace9183-d35f-431e-bb1b-d09f985d688d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:18.136688 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.136652 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ace9183-d35f-431e-bb1b-d09f985d688d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c\" (UID: \"9ace9183-d35f-431e-bb1b-d09f985d688d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:18.136750 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.136685 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee97aa7-7dcd-493c-8431-683e30c4055c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp\" (UID: \"6ee97aa7-7dcd-493c-8431-683e30c4055c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:18.238101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.238064 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvt47\" (UniqueName: \"kubernetes.io/projected/6ee97aa7-7dcd-493c-8431-683e30c4055c-kube-api-access-kvt47\") pod \"payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp\" (UID: \"6ee97aa7-7dcd-493c-8431-683e30c4055c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:18.238678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.238190 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcwk\" (UniqueName: \"kubernetes.io/projected/9ace9183-d35f-431e-bb1b-d09f985d688d-kube-api-access-vjcwk\") pod \"payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c\" (UID: \"9ace9183-d35f-431e-bb1b-d09f985d688d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:18.238678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.238342 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ace9183-d35f-431e-bb1b-d09f985d688d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c\" (UID: \"9ace9183-d35f-431e-bb1b-d09f985d688d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:18.238678 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.238408 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee97aa7-7dcd-493c-8431-683e30c4055c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp\" (UID: \"6ee97aa7-7dcd-493c-8431-683e30c4055c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:18.239445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.239415 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee97aa7-7dcd-493c-8431-683e30c4055c-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp\" (UID: \"6ee97aa7-7dcd-493c-8431-683e30c4055c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:18.240616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.240594 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ace9183-d35f-431e-bb1b-d09f985d688d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c\" (UID: \"9ace9183-d35f-431e-bb1b-d09f985d688d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:18.262526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.262314 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcwk\" (UniqueName: \"kubernetes.io/projected/9ace9183-d35f-431e-bb1b-d09f985d688d-kube-api-access-vjcwk\") pod \"payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c\" (UID: \"9ace9183-d35f-431e-bb1b-d09f985d688d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:18.262751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.262728 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvt47\" (UniqueName: \"kubernetes.io/projected/6ee97aa7-7dcd-493c-8431-683e30c4055c-kube-api-access-kvt47\") pod \"payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp\" (UID: \"6ee97aa7-7dcd-493c-8431-683e30c4055c\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:18.379403 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.379329 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:18.434412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.434381 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:18.670866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.670771 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp"] Nov 04 01:49:18.675090 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:49:18.675050 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee97aa7_7dcd_493c_8431_683e30c4055c.slice/crio-9ad0d0b94c8c066ed7b73f983478c6812f615d934593e20bf1985f6043111674 WatchSource:0}: Error finding container 9ad0d0b94c8c066ed7b73f983478c6812f615d934593e20bf1985f6043111674: Status 404 returned error can't find the container with id 9ad0d0b94c8c066ed7b73f983478c6812f615d934593e20bf1985f6043111674 Nov 04 01:49:18.742802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:18.742758 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c"] Nov 04 01:49:18.748259 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:49:18.748231 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ace9183_d35f_431e_bb1b_d09f985d688d.slice/crio-9cc152446955d2f121928db546c28c93b21f11df717d0d1132a902f85e8512f0 WatchSource:0}: Error finding container 9cc152446955d2f121928db546c28c93b21f11df717d0d1132a902f85e8512f0: Status 404 returned error can't find the container with id 9cc152446955d2f121928db546c28c93b21f11df717d0d1132a902f85e8512f0 Nov 04 01:49:19.189150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.189101 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde6cbbe-11de-4f98-b0bd-8379be9022cf" path="/var/lib/kubelet/pods/dde6cbbe-11de-4f98-b0bd-8379be9022cf/volumes" Nov 04 01:49:19.560615 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.560571 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" event={"ID":"9ace9183-d35f-431e-bb1b-d09f985d688d","Type":"ContainerStarted","Data":"d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df"} Nov 04 01:49:19.561165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.560643 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" event={"ID":"9ace9183-d35f-431e-bb1b-d09f985d688d","Type":"ContainerStarted","Data":"9cc152446955d2f121928db546c28c93b21f11df717d0d1132a902f85e8512f0"} Nov 04 01:49:19.561165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.560795 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:19.562521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.562493 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" event={"ID":"6ee97aa7-7dcd-493c-8431-683e30c4055c","Type":"ContainerStarted","Data":"7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977"} Nov 04 01:49:19.562632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.562537 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" event={"ID":"6ee97aa7-7dcd-493c-8431-683e30c4055c","Type":"ContainerStarted","Data":"9ad0d0b94c8c066ed7b73f983478c6812f615d934593e20bf1985f6043111674"} Nov 04 01:49:19.562632 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.562557 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:19.579480 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.579433 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" podStartSLOduration=1.579421735 podStartE2EDuration="1.579421735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:49:19.577741902 +0000 UTC m=+3098.549490789" watchObservedRunningTime="2026-11-04 01:49:19.579421735 +0000 UTC m=+3098.551170649" Nov 04 01:49:19.599006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:19.598941 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" podStartSLOduration=1.5989220149999999 podStartE2EDuration="1.598922015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:49:19.596486673 +0000 UTC m=+3098.568235544" watchObservedRunningTime="2026-11-04 01:49:19.598922015 +0000 UTC m=+3098.570670884" Nov 04 01:49:30.572993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:30.572911 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:49:30.572993 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:30.572978 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:49:30.642305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:30.642225 5051 generic.go:356] "Generic (PLEG): container finished" podID="b1304976-517f-4a62-99cd-9404669e2d6a" containerID="2b291820cd110798dce2f81ee1593dab15ccb3eeeb7aed2ac99b53334e38a29a" exitCode=0 Nov 04 01:49:30.642305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:30.642279 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" event={"ID":"b1304976-517f-4a62-99cd-9404669e2d6a","Type":"ContainerDied","Data":"2b291820cd110798dce2f81ee1593dab15ccb3eeeb7aed2ac99b53334e38a29a"} Nov 04 01:49:30.642655 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:30.642640 5051 scope.go:122] "RemoveContainer" containerID="2b291820cd110798dce2f81ee1593dab15ccb3eeeb7aed2ac99b53334e38a29a" Nov 04 01:49:30.643951 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:30.643921 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:49:30.644004 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:30.643994 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:49:30.644045 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:30.644027 5051 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 04 01:49:31.537273 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:31.537226 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:31.675101 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:31.675055 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" event={"ID":"b1304976-517f-4a62-99cd-9404669e2d6a","Type":"ContainerStarted","Data":"3dc756b244c64b94696c0423c0eea4c331dad368429233e66444a84e3f1822c8"} Nov 04 01:49:31.973526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:31.969507 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:31.974300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:31.974252 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 04 01:49:31.974430 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:31.974323 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 04 01:49:32.969581 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:32.969540 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 04 01:49:32.970217 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:32.970179 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 04 01:49:33.970604 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:33.970562 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 04 01:49:33.971254 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:33.971217 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 04 01:49:34.108346 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:49:34.108302 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:49:34.966147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:34.965603 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 04 01:49:34.966147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:34.965672 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 04 01:49:35.965546 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:35.965505 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 04 01:49:35.966192 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:35.966158 5051 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 04 01:49:36.970297 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:36.970264 5051 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:36.970714 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:36.970690 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:36.971536 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:36.971516 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:49:50.187943 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:49:50.187901 5051 scope.go:122] "RemoveContainer" containerID="1e9f3fad88dfab3692230a72874cfe7ef88586b5497fcef2df7c8ca4a98d2657" Nov 04 01:50:04.105649 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:50:04.105604 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:50:05.030356 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:50:05.030316 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt"] Nov 04 01:50:05.033134 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:50:05.033100 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2b6vt"] Nov 04 01:50:05.187918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:50:05.187882 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b80638f-db8e-4fe0-9f10-ad9dd670b3ed" path="/var/lib/kubelet/pods/2b80638f-db8e-4fe0-9f10-ad9dd670b3ed/volumes" Nov 04 01:50:34.106055 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:50:34.106011 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:50:57.504028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:50:57.503991 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp"] Nov 04 01:50:57.506261 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:50:57.506228 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c"] Nov 04 01:51:02.505260 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.505219 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" podUID="6ee97aa7-7dcd-493c-8431-683e30c4055c" containerName="payload-pre-processing" containerID="cri-o://7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977" gracePeriod=25 Nov 04 01:51:02.506699 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.506672 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" podUID="9ace9183-d35f-431e-bb1b-d09f985d688d" containerName="payload-processing" containerID="cri-o://d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df" gracePeriod=25 Nov 04 01:51:02.783226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.783194 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:51:02.816262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.816225 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:51:02.859412 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.859379 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ace9183-d35f-431e-bb1b-d09f985d688d-kube-api-access-vjcwk\" (UniqueName: \"kubernetes.io/projected/9ace9183-d35f-431e-bb1b-d09f985d688d-kube-api-access-vjcwk\") pod \"9ace9183-d35f-431e-bb1b-d09f985d688d\" (UID: \"9ace9183-d35f-431e-bb1b-d09f985d688d\") " Nov 04 01:51:02.859604 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.859456 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ace9183-d35f-431e-bb1b-d09f985d688d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ace9183-d35f-431e-bb1b-d09f985d688d-plugins-config-volume\") pod \"9ace9183-d35f-431e-bb1b-d09f985d688d\" (UID: \"9ace9183-d35f-431e-bb1b-d09f985d688d\") " Nov 04 01:51:02.859916 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.859891 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ace9183-d35f-431e-bb1b-d09f985d688d-plugins-config-volume" pod "9ace9183-d35f-431e-bb1b-d09f985d688d" (UID: "9ace9183-d35f-431e-bb1b-d09f985d688d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:02.861619 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.861598 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ace9183-d35f-431e-bb1b-d09f985d688d-kube-api-access-vjcwk" pod "9ace9183-d35f-431e-bb1b-d09f985d688d" (UID: "9ace9183-d35f-431e-bb1b-d09f985d688d"). InnerVolumeSpecName "kube-api-access-vjcwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:02.960280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.960248 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee97aa7-7dcd-493c-8431-683e30c4055c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee97aa7-7dcd-493c-8431-683e30c4055c-plugins-config-volume\") pod \"6ee97aa7-7dcd-493c-8431-683e30c4055c\" (UID: \"6ee97aa7-7dcd-493c-8431-683e30c4055c\") " Nov 04 01:51:02.960447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.960363 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee97aa7-7dcd-493c-8431-683e30c4055c-kube-api-access-kvt47\" (UniqueName: \"kubernetes.io/projected/6ee97aa7-7dcd-493c-8431-683e30c4055c-kube-api-access-kvt47\") pod \"6ee97aa7-7dcd-493c-8431-683e30c4055c\" (UID: \"6ee97aa7-7dcd-493c-8431-683e30c4055c\") " Nov 04 01:51:02.960617 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.960603 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjcwk\" (UniqueName: \"kubernetes.io/projected/9ace9183-d35f-431e-bb1b-d09f985d688d-kube-api-access-vjcwk\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:02.960666 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.960621 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ace9183-d35f-431e-bb1b-d09f985d688d-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:02.960711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.960682 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee97aa7-7dcd-493c-8431-683e30c4055c-plugins-config-volume" pod "6ee97aa7-7dcd-493c-8431-683e30c4055c" (UID: "6ee97aa7-7dcd-493c-8431-683e30c4055c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:02.962519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:02.962497 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee97aa7-7dcd-493c-8431-683e30c4055c-kube-api-access-kvt47" pod "6ee97aa7-7dcd-493c-8431-683e30c4055c" (UID: "6ee97aa7-7dcd-493c-8431-683e30c4055c"). InnerVolumeSpecName "kube-api-access-kvt47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:03.061481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.061382 5051 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee97aa7-7dcd-493c-8431-683e30c4055c-plugins-config-volume\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:03.061481 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.061419 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvt47\" (UniqueName: \"kubernetes.io/projected/6ee97aa7-7dcd-493c-8431-683e30c4055c-kube-api-access-kvt47\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:03.197394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.197362 5051 generic.go:356] "Generic (PLEG): container finished" podID="9ace9183-d35f-431e-bb1b-d09f985d688d" containerID="d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df" exitCode=0 Nov 04 01:51:03.197555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.197403 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" Nov 04 01:51:03.197555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.197449 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" event={"ID":"9ace9183-d35f-431e-bb1b-d09f985d688d","Type":"ContainerDied","Data":"d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df"} Nov 04 01:51:03.197555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.197507 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c" event={"ID":"9ace9183-d35f-431e-bb1b-d09f985d688d","Type":"ContainerDied","Data":"9cc152446955d2f121928db546c28c93b21f11df717d0d1132a902f85e8512f0"} Nov 04 01:51:03.197555 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.197538 5051 scope.go:122] "RemoveContainer" containerID="d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df" Nov 04 01:51:03.199435 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.199407 5051 generic.go:356] "Generic (PLEG): container finished" podID="6ee97aa7-7dcd-493c-8431-683e30c4055c" containerID="7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977" exitCode=0 Nov 04 01:51:03.199534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.199442 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" Nov 04 01:51:03.199534 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.199496 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" event={"ID":"6ee97aa7-7dcd-493c-8431-683e30c4055c","Type":"ContainerDied","Data":"7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977"} Nov 04 01:51:03.199635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.199540 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp" event={"ID":"6ee97aa7-7dcd-493c-8431-683e30c4055c","Type":"ContainerDied","Data":"9ad0d0b94c8c066ed7b73f983478c6812f615d934593e20bf1985f6043111674"} Nov 04 01:51:03.208643 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.208626 5051 scope.go:122] "RemoveContainer" containerID="d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df" Nov 04 01:51:03.208914 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:51:03.208897 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df\": container with ID starting with d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df not found: ID does not exist" containerID="d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df" Nov 04 01:51:03.208965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.208921 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df"} err="failed to get container status \"d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df\": rpc error: code = NotFound desc = could not find container \"d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df\": container with ID starting with d1c36ea30803469836487eae6ef27a48a753929c300ac3fb0513cf556d3ff6df not found: ID does not exist" Nov 04 01:51:03.208965 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.208937 5051 scope.go:122] "RemoveContainer" containerID="7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977" Nov 04 01:51:03.217344 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.217328 5051 scope.go:122] "RemoveContainer" containerID="7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977" Nov 04 01:51:03.217595 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:51:03.217572 5051 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977\": container with ID starting with 7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977 not found: ID does not exist" containerID="7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977" Nov 04 01:51:03.217676 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.217604 5051 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977"} err="failed to get container status \"7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977\": rpc error: code = NotFound desc = could not find container \"7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977\": container with ID starting with 7bc445f71a2f867b4fb1d577bcb9a11e384f304dccb70743a52dfe70a4091977 not found: ID does not exist" Nov 04 01:51:03.218104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.218081 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp"] Nov 04 01:51:03.221036 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.221014 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-442e8c-7665b9cbd7-84lsp"] Nov 04 01:51:03.230736 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.230711 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c"] Nov 04 01:51:03.232984 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:03.232962 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-442e8c-68b9fd5476-x544c"] Nov 04 01:51:04.106022 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:51:04.105986 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:51:05.102970 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:05.102936 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb"] Nov 04 01:51:05.103258 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:05.103137 5051 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" containerID="cri-o://3dc756b244c64b94696c0423c0eea4c331dad368429233e66444a84e3f1822c8" gracePeriod=30 Nov 04 01:51:05.187954 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:05.187919 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee97aa7-7dcd-493c-8431-683e30c4055c" path="/var/lib/kubelet/pods/6ee97aa7-7dcd-493c-8431-683e30c4055c/volumes" Nov 04 01:51:05.188362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:05.188349 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ace9183-d35f-431e-bb1b-d09f985d688d" path="/var/lib/kubelet/pods/9ace9183-d35f-431e-bb1b-d09f985d688d/volumes" Nov 04 01:51:06.973214 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:06.973178 5051 patch_prober.go:29] interesting pod/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 01:51:06.973624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:06.973238 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 01:51:07.020849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:07.020813 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4"] Nov 04 01:51:07.023477 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:07.023448 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b35766fe-jskt4"] Nov 04 01:51:07.187711 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:07.187677 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f313598-269b-4ac5-b6de-c141f56f20b5" path="/var/lib/kubelet/pods/0f313598-269b-4ac5-b6de-c141f56f20b5/volumes" Nov 04 01:51:10.327076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:10.326806 5051 generic.go:356] "Generic (PLEG): container finished" podID="b1304976-517f-4a62-99cd-9404669e2d6a" containerID="3dc756b244c64b94696c0423c0eea4c331dad368429233e66444a84e3f1822c8" exitCode=0 Nov 04 01:51:10.327076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:10.326913 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" event={"ID":"b1304976-517f-4a62-99cd-9404669e2d6a","Type":"ContainerDied","Data":"3dc756b244c64b94696c0423c0eea4c331dad368429233e66444a84e3f1822c8"} Nov 04 01:51:10.327076 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:10.326975 5051 scope.go:122] "RemoveContainer" containerID="2b291820cd110798dce2f81ee1593dab15ccb3eeeb7aed2ac99b53334e38a29a" Nov 04 01:51:11.114104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.113289 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xfhf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:51:11.114104 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.113382 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xfhf7" podUID="01677e74-8528-41df-aa1a-947aefe19645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:51:11.114519 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.114489 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-xkmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:51:11.153501 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.153355 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-xkmtt" podUID="ed9a741a-6c41-4981-a205-f01665ad111e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:51:11.248810 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.248733 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:51:11.365002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.364705 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" event={"ID":"b1304976-517f-4a62-99cd-9404669e2d6a","Type":"ContainerDied","Data":"7825d8635abcd4b85e3416d9207eec642063724e86dbcb89460dc983671abc46"} Nov 04 01:51:11.365755 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.365717 5051 scope.go:122] "RemoveContainer" containerID="3dc756b244c64b94696c0423c0eea4c331dad368429233e66444a84e3f1822c8" Nov 04 01:51:11.366226 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.366193 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb" Nov 04 01:51:11.379391 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.379362 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-credential-socket\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.381830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.381810 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-envoy\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.382394 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.382374 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-socket\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.382835 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.382815 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-data\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.383352 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.383334 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-certs\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.383855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.383833 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-kube-api-access-g8zrm\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-kube-api-access-g8zrm\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.384381 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.384361 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1304976-517f-4a62-99cd-9404669e2d6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1304976-517f-4a62-99cd-9404669e2d6a-istiod-ca-cert\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.385002 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.384984 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1304976-517f-4a62-99cd-9404669e2d6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1304976-517f-4a62-99cd-9404669e2d6a-istio-podinfo\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.399789 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.399764 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-istio-token\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-istio-token\") pod \"b1304976-517f-4a62-99cd-9404669e2d6a\" (UID: \"b1304976-517f-4a62-99cd-9404669e2d6a\") " Nov 04 01:51:11.400870 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.381725 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-credential-socket" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:51:11.401016 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.382729 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-socket" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:51:11.401165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.383186 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-data" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:51:11.401272 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.383750 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-certs" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:51:11.401384 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.384833 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1304976-517f-4a62-99cd-9404669e2d6a-istiod-ca-cert" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:11.406471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.406312 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-envoy" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 01:51:11.418649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.418157 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-kube-api-access-g8zrm" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "kube-api-access-g8zrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:11.418649 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.418554 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1304976-517f-4a62-99cd-9404669e2d6a-istio-podinfo" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 01:51:11.420234 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.420204 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-istio-token" pod "b1304976-517f-4a62-99cd-9404669e2d6a" (UID: "b1304976-517f-4a62-99cd-9404669e2d6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:11.501628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501529 5051 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-data\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.501628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501572 5051 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-certs\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.501628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501586 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8zrm\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-kube-api-access-g8zrm\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.501628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501596 5051 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1304976-517f-4a62-99cd-9404669e2d6a-istiod-ca-cert\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.501628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501608 5051 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1304976-517f-4a62-99cd-9404669e2d6a-istio-podinfo\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.501628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501622 5051 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1304976-517f-4a62-99cd-9404669e2d6a-istio-token\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.501628 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501631 5051 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-credential-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.502175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501640 5051 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-istio-envoy\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.502175 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.501648 5051 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1304976-517f-4a62-99cd-9404669e2d6a-workload-socket\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:11.694032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.693996 5051 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb"] Nov 04 01:51:11.698010 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:11.697976 5051 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-442e8c-openshift-default-c889cf8d9-mm9kb"] Nov 04 01:51:13.188802 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:13.188760 5051 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" path="/var/lib/kubelet/pods/b1304976-517f-4a62-99cd-9404669e2d6a/volumes" Nov 04 01:51:18.558742 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.558704 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz"] Nov 04 01:51:18.559169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559151 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" Nov 04 01:51:18.559169 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559165 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" Nov 04 01:51:18.559262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559180 5051 cpu_manager.go:414] "containerMap: removing container" podUID="9ace9183-d35f-431e-bb1b-d09f985d688d" containerName="payload-processing" Nov 04 01:51:18.559262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559185 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ace9183-d35f-431e-bb1b-d09f985d688d" containerName="payload-processing" Nov 04 01:51:18.559262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559202 5051 cpu_manager.go:414] "containerMap: removing container" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" Nov 04 01:51:18.559262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559207 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" Nov 04 01:51:18.559262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559229 5051 cpu_manager.go:414] "containerMap: removing container" podUID="6ee97aa7-7dcd-493c-8431-683e30c4055c" containerName="payload-pre-processing" Nov 04 01:51:18.559262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559234 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee97aa7-7dcd-493c-8431-683e30c4055c" containerName="payload-pre-processing" Nov 04 01:51:18.559449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559305 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" Nov 04 01:51:18.559449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559314 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ace9183-d35f-431e-bb1b-d09f985d688d" containerName="payload-processing" Nov 04 01:51:18.559449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559322 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1304976-517f-4a62-99cd-9404669e2d6a" containerName="istio-proxy" Nov 04 01:51:18.559449 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.559333 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee97aa7-7dcd-493c-8431-683e30c4055c" containerName="payload-pre-processing" Nov 04 01:51:18.562554 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.562532 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.568336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.567796 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:51:18.568336 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.567858 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:51:18.568962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.568918 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2jjz8\"" Nov 04 01:51:18.568962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.568919 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:51:18.572419 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.572401 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz"] Nov 04 01:51:18.625493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.625461 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr67s\" (UniqueName: \"kubernetes.io/projected/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-kube-api-access-nr67s\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.625665 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.625642 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.625734 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.625679 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.726849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.726812 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.726849 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.726855 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.727110 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.726896 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr67s\" (UniqueName: \"kubernetes.io/projected/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-kube-api-access-nr67s\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.727686 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.727655 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.729407 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.729380 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.734834 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.734804 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr67s\" (UniqueName: \"kubernetes.io/projected/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-kube-api-access-nr67s\") pod \"maas-api-revoke-sub-f03629db-z9rmz\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:18.889988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:18.889897 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:19.168485 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:19.168452 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz"] Nov 04 01:51:19.171318 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:51:19.171287 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b23daf_a5e6_499b_bc32_1ec05d1235bc.slice/crio-af291a1421ee8bf7a9e6bd54d3b17d115e777b80159151a5f9dd498c56562735 WatchSource:0}: Error finding container af291a1421ee8bf7a9e6bd54d3b17d115e777b80159151a5f9dd498c56562735: Status 404 returned error can't find the container with id af291a1421ee8bf7a9e6bd54d3b17d115e777b80159151a5f9dd498c56562735 Nov 04 01:51:19.454135 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:19.454086 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" event={"ID":"a8b23daf-a5e6-499b-bc32-1ec05d1235bc","Type":"ContainerStarted","Data":"af291a1421ee8bf7a9e6bd54d3b17d115e777b80159151a5f9dd498c56562735"} Nov 04 01:51:20.461749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:20.461703 5051 generic.go:356] "Generic (PLEG): container finished" podID="a8b23daf-a5e6-499b-bc32-1ec05d1235bc" containerID="3ba1aab2c41829093c57683a33667917ed6f187cf3905e29b4e035c7cb29f077" exitCode=0 Nov 04 01:51:20.461749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:20.461752 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" event={"ID":"a8b23daf-a5e6-499b-bc32-1ec05d1235bc","Type":"ContainerDied","Data":"3ba1aab2c41829093c57683a33667917ed6f187cf3905e29b4e035c7cb29f077"} Nov 04 01:51:21.610831 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.610807 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:21.758630 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.758590 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-openshift-ca\") pod \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " Nov 04 01:51:21.758811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.758645 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-xks-ca\") pod \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " Nov 04 01:51:21.758811 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.758670 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-kube-api-access-nr67s\" (UniqueName: \"kubernetes.io/projected/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-kube-api-access-nr67s\") pod \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\" (UID: \"a8b23daf-a5e6-499b-bc32-1ec05d1235bc\") " Nov 04 01:51:21.758966 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.758918 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-openshift-ca" pod "a8b23daf-a5e6-499b-bc32-1ec05d1235bc" (UID: "a8b23daf-a5e6-499b-bc32-1ec05d1235bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:21.759051 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.759028 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:21.760866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.760836 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-xks-ca" pod "a8b23daf-a5e6-499b-bc32-1ec05d1235bc" (UID: "a8b23daf-a5e6-499b-bc32-1ec05d1235bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:51:21.760866 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.760851 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-kube-api-access-nr67s" pod "a8b23daf-a5e6-499b-bc32-1ec05d1235bc" (UID: "a8b23daf-a5e6-499b-bc32-1ec05d1235bc"). InnerVolumeSpecName "kube-api-access-nr67s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:21.860248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.860224 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:21.860248 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:21.860251 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr67s\" (UniqueName: \"kubernetes.io/projected/a8b23daf-a5e6-499b-bc32-1ec05d1235bc-kube-api-access-nr67s\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:22.473314 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:22.473287 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" Nov 04 01:51:22.473489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:22.473281 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z9rmz" event={"ID":"a8b23daf-a5e6-499b-bc32-1ec05d1235bc","Type":"ContainerDied","Data":"af291a1421ee8bf7a9e6bd54d3b17d115e777b80159151a5f9dd498c56562735"} Nov 04 01:51:22.473489 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:22.473413 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af291a1421ee8bf7a9e6bd54d3b17d115e777b80159151a5f9dd498c56562735" Nov 04 01:51:33.969144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:33.968230 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bk4g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:51:33.969144 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:33.968301 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bk4g2" podUID="8d8804da-7930-4bee-9769-6910cf9db442" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:51:34.256396 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:51:34.246536 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:51:34.256396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:34.248209 5051 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2mvtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 04 01:51:34.256396 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:34.248264 5051 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2mvtq" podUID="5f694985-9c0e-45ab-93b4-fa5ebe379290" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 04 01:51:35.217928 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.217887 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv"] Nov 04 01:51:35.218364 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.218348 5051 cpu_manager.go:414] "containerMap: removing container" podUID="a8b23daf-a5e6-499b-bc32-1ec05d1235bc" containerName="revoke-keys" Nov 04 01:51:35.218415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.218365 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b23daf-a5e6-499b-bc32-1ec05d1235bc" containerName="revoke-keys" Nov 04 01:51:35.218454 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.218446 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b23daf-a5e6-499b-bc32-1ec05d1235bc" containerName="revoke-keys" Nov 04 01:51:35.221769 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.221750 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.229365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.229336 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:51:35.229558 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.229339 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:51:35.230847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.229372 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:51:35.230847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.229524 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2jjz8\"" Nov 04 01:51:35.233065 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.233037 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv"] Nov 04 01:51:35.346830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.346779 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49sc6\" (UniqueName: \"kubernetes.io/projected/7afc209f-ea37-4b81-a741-55b5e57a4126-kube-api-access-49sc6\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.347028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.346864 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.347028 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.346983 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.447780 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.447731 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.447988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.447844 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49sc6\" (UniqueName: \"kubernetes.io/projected/7afc209f-ea37-4b81-a741-55b5e57a4126-kube-api-access-49sc6\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.447988 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.447917 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.448512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.448479 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.450406 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.450382 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.458056 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.458028 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49sc6\" (UniqueName: \"kubernetes.io/projected/7afc209f-ea37-4b81-a741-55b5e57a4126-kube-api-access-49sc6\") pod \"maas-api-revoke-sub-0165d7d2-295nv\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.539116 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.539080 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:35.816918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:35.816893 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv"] Nov 04 01:51:35.823026 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:51:35.822996 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afc209f_ea37_4b81_a741_55b5e57a4126.slice/crio-ea912fcd2d962a814718e72f6a7266c33a9a04832bdba931fbb1dd9025c4310d WatchSource:0}: Error finding container ea912fcd2d962a814718e72f6a7266c33a9a04832bdba931fbb1dd9025c4310d: Status 404 returned error can't find the container with id ea912fcd2d962a814718e72f6a7266c33a9a04832bdba931fbb1dd9025c4310d Nov 04 01:51:36.607523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:36.607484 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" event={"ID":"7afc209f-ea37-4b81-a741-55b5e57a4126","Type":"ContainerStarted","Data":"54db03ea03f19a3fdba14a3045c75c2e1f8d533dcb9b1d0aae7ab7866313c869"} Nov 04 01:51:36.607906 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:36.607548 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" event={"ID":"7afc209f-ea37-4b81-a741-55b5e57a4126","Type":"ContainerStarted","Data":"ea912fcd2d962a814718e72f6a7266c33a9a04832bdba931fbb1dd9025c4310d"} Nov 04 01:51:36.626703 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:36.626648 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" podStartSLOduration=1.626633328 podStartE2EDuration="1.626633328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:51:36.625152448 +0000 UTC m=+3235.596901316" watchObservedRunningTime="2026-11-04 01:51:36.626633328 +0000 UTC m=+3235.598382199" Nov 04 01:51:37.615328 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:37.615292 5051 generic.go:356] "Generic (PLEG): container finished" podID="7afc209f-ea37-4b81-a741-55b5e57a4126" containerID="54db03ea03f19a3fdba14a3045c75c2e1f8d533dcb9b1d0aae7ab7866313c869" exitCode=0 Nov 04 01:51:37.615758 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:37.615386 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" event={"ID":"7afc209f-ea37-4b81-a741-55b5e57a4126","Type":"ContainerDied","Data":"54db03ea03f19a3fdba14a3045c75c2e1f8d533dcb9b1d0aae7ab7866313c869"} Nov 04 01:51:38.766917 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.766888 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:38.887300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.887222 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-openshift-ca\") pod \"7afc209f-ea37-4b81-a741-55b5e57a4126\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " Nov 04 01:51:38.887300 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.887273 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-xks-ca\") pod \"7afc209f-ea37-4b81-a741-55b5e57a4126\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " Nov 04 01:51:38.887488 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.887313 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afc209f-ea37-4b81-a741-55b5e57a4126-kube-api-access-49sc6\" (UniqueName: \"kubernetes.io/projected/7afc209f-ea37-4b81-a741-55b5e57a4126-kube-api-access-49sc6\") pod \"7afc209f-ea37-4b81-a741-55b5e57a4126\" (UID: \"7afc209f-ea37-4b81-a741-55b5e57a4126\") " Nov 04 01:51:38.887663 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.887641 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-openshift-ca" pod "7afc209f-ea37-4b81-a741-55b5e57a4126" (UID: "7afc209f-ea37-4b81-a741-55b5e57a4126"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:38.887781 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.887760 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:38.889503 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.889477 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-xks-ca" pod "7afc209f-ea37-4b81-a741-55b5e57a4126" (UID: "7afc209f-ea37-4b81-a741-55b5e57a4126"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:51:38.889602 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.889554 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afc209f-ea37-4b81-a741-55b5e57a4126-kube-api-access-49sc6" pod "7afc209f-ea37-4b81-a741-55b5e57a4126" (UID: "7afc209f-ea37-4b81-a741-55b5e57a4126"). InnerVolumeSpecName "kube-api-access-49sc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:38.988523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.988483 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7afc209f-ea37-4b81-a741-55b5e57a4126-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:38.988523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:38.988515 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49sc6\" (UniqueName: \"kubernetes.io/projected/7afc209f-ea37-4b81-a741-55b5e57a4126-kube-api-access-49sc6\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:39.628493 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:39.628458 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" event={"ID":"7afc209f-ea37-4b81-a741-55b5e57a4126","Type":"ContainerDied","Data":"ea912fcd2d962a814718e72f6a7266c33a9a04832bdba931fbb1dd9025c4310d"} Nov 04 01:51:39.628674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:39.628511 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea912fcd2d962a814718e72f6a7266c33a9a04832bdba931fbb1dd9025c4310d" Nov 04 01:51:39.628674 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:39.628471 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-295nv" Nov 04 01:51:45.909792 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.909757 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t"] Nov 04 01:51:45.910268 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.910251 5051 cpu_manager.go:414] "containerMap: removing container" podUID="7afc209f-ea37-4b81-a741-55b5e57a4126" containerName="revoke-keys" Nov 04 01:51:45.910321 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.910270 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7afc209f-ea37-4b81-a741-55b5e57a4126" containerName="revoke-keys" Nov 04 01:51:45.910367 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.910350 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="7afc209f-ea37-4b81-a741-55b5e57a4126" containerName="revoke-keys" Nov 04 01:51:45.913601 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.913584 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:45.918451 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.918387 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 01:51:45.918613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.918549 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 01:51:45.919855 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.919833 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2jjz8\"" Nov 04 01:51:45.920031 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.920018 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 01:51:45.921830 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.921682 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t"] Nov 04 01:51:45.948280 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.948248 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:45.948463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.948311 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tr2\" (UniqueName: \"kubernetes.io/projected/dbac0115-78f9-4789-8073-2a202b93a9e6-kube-api-access-d6tr2\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:45.948463 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:45.948446 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:46.049434 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.049394 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:46.049624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.049463 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tr2\" (UniqueName: \"kubernetes.io/projected/dbac0115-78f9-4789-8073-2a202b93a9e6-kube-api-access-d6tr2\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:46.049624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.049495 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:46.050262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.050229 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:46.051913 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.051885 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:46.057446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.057421 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tr2\" (UniqueName: \"kubernetes.io/projected/dbac0115-78f9-4789-8073-2a202b93a9e6-kube-api-access-d6tr2\") pod \"maas-api-revoke-sub-6b19338f-rlr4t\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:46.229664 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.229569 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:46.515506 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.515371 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t"] Nov 04 01:51:46.518528 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:51:46.518498 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbac0115_78f9_4789_8073_2a202b93a9e6.slice/crio-5dbf7884ad87e4d06866ec39abf78098d6b224b3c1689b6e217eedbb10ea9532 WatchSource:0}: Error finding container 5dbf7884ad87e4d06866ec39abf78098d6b224b3c1689b6e217eedbb10ea9532: Status 404 returned error can't find the container with id 5dbf7884ad87e4d06866ec39abf78098d6b224b3c1689b6e217eedbb10ea9532 Nov 04 01:51:46.668922 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:46.668882 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" event={"ID":"dbac0115-78f9-4789-8073-2a202b93a9e6","Type":"ContainerStarted","Data":"5dbf7884ad87e4d06866ec39abf78098d6b224b3c1689b6e217eedbb10ea9532"} Nov 04 01:51:47.676095 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:47.676057 5051 generic.go:356] "Generic (PLEG): container finished" podID="dbac0115-78f9-4789-8073-2a202b93a9e6" containerID="1a2b5fa21a39eb60f9a0f3248da9def765a08be03e60131fecbc638814088c47" exitCode=0 Nov 04 01:51:47.676526 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:47.676111 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" event={"ID":"dbac0115-78f9-4789-8073-2a202b93a9e6","Type":"ContainerDied","Data":"1a2b5fa21a39eb60f9a0f3248da9def765a08be03e60131fecbc638814088c47"} Nov 04 01:51:48.817752 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.817726 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:48.879235 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.879202 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbac0115-78f9-4789-8073-2a202b93a9e6-kube-api-access-d6tr2\" (UniqueName: \"kubernetes.io/projected/dbac0115-78f9-4789-8073-2a202b93a9e6-kube-api-access-d6tr2\") pod \"dbac0115-78f9-4789-8073-2a202b93a9e6\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " Nov 04 01:51:48.879415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.879307 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-openshift-ca\") pod \"dbac0115-78f9-4789-8073-2a202b93a9e6\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " Nov 04 01:51:48.879415 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.879359 5051 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-xks-ca\") pod \"dbac0115-78f9-4789-8073-2a202b93a9e6\" (UID: \"dbac0115-78f9-4789-8073-2a202b93a9e6\") " Nov 04 01:51:48.879647 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.879623 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-openshift-ca" pod "dbac0115-78f9-4789-8073-2a202b93a9e6" (UID: "dbac0115-78f9-4789-8073-2a202b93a9e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 01:51:48.881523 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.881493 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbac0115-78f9-4789-8073-2a202b93a9e6-kube-api-access-d6tr2" pod "dbac0115-78f9-4789-8073-2a202b93a9e6" (UID: "dbac0115-78f9-4789-8073-2a202b93a9e6"). InnerVolumeSpecName "kube-api-access-d6tr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 01:51:48.881659 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.881559 5051 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-xks-ca" pod "dbac0115-78f9-4789-8073-2a202b93a9e6" (UID: "dbac0115-78f9-4789-8073-2a202b93a9e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 01:51:48.980727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.980632 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-openshift-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:48.980727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.980669 5051 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac0115-78f9-4789-8073-2a202b93a9e6-maas-api-xks-ca\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:48.980727 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:48.980681 5051 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6tr2\" (UniqueName: \"kubernetes.io/projected/dbac0115-78f9-4789-8073-2a202b93a9e6-kube-api-access-d6tr2\") on node \"ip-10-0-137-48.us-east-2.compute.internal\" DevicePath \"\"" Nov 04 01:51:49.688164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:49.688097 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" event={"ID":"dbac0115-78f9-4789-8073-2a202b93a9e6","Type":"ContainerDied","Data":"5dbf7884ad87e4d06866ec39abf78098d6b224b3c1689b6e217eedbb10ea9532"} Nov 04 01:51:49.688164 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:49.688159 5051 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rlr4t" Nov 04 01:51:49.688433 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:49.688186 5051 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dbf7884ad87e4d06866ec39abf78098d6b224b3c1689b6e217eedbb10ea9532" Nov 04 01:51:50.232543 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:50.232509 5051 scope.go:122] "RemoveContainer" containerID="ea7eef5bdee284f89fbb5250bcb0055ab4bcf7ac072b21bdc078d970bea6bd89" Nov 04 01:51:57.149365 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:51:57.149327 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7457bc8fc9-m4sb2_d9622120-9f16-41e2-8a70-77f05616c703/authorino/0.log" Nov 04 01:52:04.105653 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:52:04.105607 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:52:09.284262 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:09.284227 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ph6rn_11a8492c-3577-4e3c-b7ab-a8f84b5d8b03/manager/0.log" Nov 04 01:52:11.730275 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:11.730243 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-295nv_7afc209f-ea37-4b81-a741-55b5e57a4126/revoke-keys/0.log" Nov 04 01:52:12.430538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:12.430501 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-rlr4t_dbac0115-78f9-4789-8073-2a202b93a9e6/revoke-keys/0.log" Nov 04 01:52:12.597798 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:12.597767 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-z9rmz_a8b23daf-a5e6-499b-bc32-1ec05d1235bc/revoke-keys/0.log" Nov 04 01:52:14.602305 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:14.602270 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7457bc8fc9-m4sb2_d9622120-9f16-41e2-8a70-77f05616c703/authorino/0.log" Nov 04 01:52:15.009438 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:15.009403 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tmmwg_c9f5857b-b55b-4002-b4a3-4f61950bd6ba/manager/1.log" Nov 04 01:52:15.445848 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:15.445761 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rztrb_f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87/manager/0.log" Nov 04 01:52:16.442578 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:16.442542 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tmmwg_c9f5857b-b55b-4002-b4a3-4f61950bd6ba/manager/0.log" Nov 04 01:52:16.994903 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:16.994865 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6_6d63414f-bab7-4ddc-b3fe-aa0da66e7881/istio-proxy/0.log" Nov 04 01:52:17.690342 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:17.690307 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2mvtq_5f694985-9c0e-45ab-93b4-fa5ebe379290/istio-proxy/0.log" Nov 04 01:52:18.234891 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:18.234854 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bk4g2_8d8804da-7930-4bee-9769-6910cf9db442/istio-proxy/0.log" Nov 04 01:52:19.166805 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:19.166775 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/1.log" Nov 04 01:52:19.346298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:19.346266 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xkmtt_ed9a741a-6c41-4981-a205-f01665ad111e/istio-proxy/0.log" Nov 04 01:52:20.042737 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:20.042675 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-669749fcf7-vqftg_7d024224-166d-4cb4-976f-75e0b47addae/router/0.log" Nov 04 01:52:20.631846 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:20.631814 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/0.log" Nov 04 01:52:21.128733 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:21.128703 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s_56a1d019-a504-4116-929d-33ad787904e5/main/0.log" Nov 04 01:52:21.679267 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:21.679191 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-g4dbj_51487e32-bffb-43e9-a770-720b3d6d383c/main/0.log" Nov 04 01:52:22.048165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:22.048133 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-kt9vs_f80b3f88-24db-4e50-aecc-c1c19664e0ae/main/0.log" Nov 04 01:52:22.235899 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:22.235864 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p_e1f2221e-2cb7-4513-a72c-e6e4156d6b73/main/0.log" Nov 04 01:52:23.514378 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:23.514345 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7457bc8fc9-m4sb2_d9622120-9f16-41e2-8a70-77f05616c703/authorino/0.log" Nov 04 01:52:33.000492 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:33.000458 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ph6rn_11a8492c-3577-4e3c-b7ab-a8f84b5d8b03/manager/0.log" Nov 04 01:52:34.104991 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:52:34.104950 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:52:34.924561 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:34.924538 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-295nv_7afc209f-ea37-4b81-a741-55b5e57a4126/revoke-keys/0.log" Nov 04 01:52:35.652242 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:35.652214 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-rlr4t_dbac0115-78f9-4789-8073-2a202b93a9e6/revoke-keys/0.log" Nov 04 01:52:35.831240 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:35.831203 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-z9rmz_a8b23daf-a5e6-499b-bc32-1ec05d1235bc/revoke-keys/0.log" Nov 04 01:52:37.182377 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:37.182342 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7457bc8fc9-m4sb2_d9622120-9f16-41e2-8a70-77f05616c703/authorino/0.log" Nov 04 01:52:37.584233 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:37.584206 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tmmwg_c9f5857b-b55b-4002-b4a3-4f61950bd6ba/manager/1.log" Nov 04 01:52:37.971613 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:37.971532 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rztrb_f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87/manager/0.log" Nov 04 01:52:38.940693 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:38.940661 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6_6d63414f-bab7-4ddc-b3fe-aa0da66e7881/istio-proxy/0.log" Nov 04 01:52:39.691157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:39.691108 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2mvtq_5f694985-9c0e-45ab-93b4-fa5ebe379290/istio-proxy/0.log" Nov 04 01:52:40.296853 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:40.296820 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bk4g2_8d8804da-7930-4bee-9769-6910cf9db442/istio-proxy/0.log" Nov 04 01:52:41.308170 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:41.308108 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/1.log" Nov 04 01:52:41.484167 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:41.484097 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xkmtt_ed9a741a-6c41-4981-a205-f01665ad111e/istio-proxy/0.log" Nov 04 01:52:42.259143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:42.259091 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-669749fcf7-vqftg_7d024224-166d-4cb4-976f-75e0b47addae/router/0.log" Nov 04 01:52:42.821879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:42.821849 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qg27s_56a1d019-a504-4116-929d-33ad787904e5/main/0.log" Nov 04 01:52:43.231864 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:43.231788 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:52:43.263976 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:43.263945 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:52:43.374723 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:43.374690 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-g4dbj_51487e32-bffb-43e9-a770-720b3d6d383c/main/0.log" Nov 04 01:52:43.762514 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:43.762477 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-kt9vs_f80b3f88-24db-4e50-aecc-c1c19664e0ae/main/0.log" Nov 04 01:52:43.955318 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:43.955281 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-tm85p_e1f2221e-2cb7-4513-a72c-e6e4156d6b73/main/0.log" Nov 04 01:52:50.270298 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:52:50.270260 5051 scope.go:122] "RemoveContainer" containerID="233380d7515df37111e2633bca302a0c5863ed1aa530e8b2fcf118a7021404ef" Nov 04 01:53:04.104831 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:53:04.104790 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:53:34.104648 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:53:34.104609 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:53:49.537112 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.537078 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pplq5/must-gather-jhqwd"] Nov 04 01:53:49.537536 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.537521 5051 cpu_manager.go:414] "containerMap: removing container" podUID="dbac0115-78f9-4789-8073-2a202b93a9e6" containerName="revoke-keys" Nov 04 01:53:49.537582 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.537537 5051 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbac0115-78f9-4789-8073-2a202b93a9e6" containerName="revoke-keys" Nov 04 01:53:49.537651 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.537642 5051 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbac0115-78f9-4789-8073-2a202b93a9e6" containerName="revoke-keys" Nov 04 01:53:49.540955 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.540938 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pplq5/must-gather-jhqwd" Nov 04 01:53:49.545256 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.545229 5051 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pplq5\"/\"default-dockercfg-lhzcx\"" Nov 04 01:53:49.545385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.545292 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pplq5\"/\"openshift-service-ca.crt\"" Nov 04 01:53:49.545385 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.545295 5051 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pplq5\"/\"kube-root-ca.crt\"" Nov 04 01:53:49.590246 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.590219 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pplq5/must-gather-jhqwd"] Nov 04 01:53:49.595840 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.595813 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9kvq\" (UniqueName: \"kubernetes.io/projected/d49c1137-0305-4431-9e94-37ef9c03616a-kube-api-access-q9kvq\") pod \"must-gather-jhqwd\" (UID: \"d49c1137-0305-4431-9e94-37ef9c03616a\") " pod="openshift-must-gather-pplq5/must-gather-jhqwd" Nov 04 01:53:49.595942 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.595890 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d49c1137-0305-4431-9e94-37ef9c03616a-must-gather-output\") pod \"must-gather-jhqwd\" (UID: \"d49c1137-0305-4431-9e94-37ef9c03616a\") " pod="openshift-must-gather-pplq5/must-gather-jhqwd" Nov 04 01:53:49.696813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.696781 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kvq\" (UniqueName: \"kubernetes.io/projected/d49c1137-0305-4431-9e94-37ef9c03616a-kube-api-access-q9kvq\") pod \"must-gather-jhqwd\" (UID: \"d49c1137-0305-4431-9e94-37ef9c03616a\") " pod="openshift-must-gather-pplq5/must-gather-jhqwd" Nov 04 01:53:49.696962 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.696834 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d49c1137-0305-4431-9e94-37ef9c03616a-must-gather-output\") pod \"must-gather-jhqwd\" (UID: \"d49c1137-0305-4431-9e94-37ef9c03616a\") " pod="openshift-must-gather-pplq5/must-gather-jhqwd" Nov 04 01:53:49.697187 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.697170 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d49c1137-0305-4431-9e94-37ef9c03616a-must-gather-output\") pod \"must-gather-jhqwd\" (UID: \"d49c1137-0305-4431-9e94-37ef9c03616a\") " pod="openshift-must-gather-pplq5/must-gather-jhqwd" Nov 04 01:53:49.705836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.705815 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kvq\" (UniqueName: \"kubernetes.io/projected/d49c1137-0305-4431-9e94-37ef9c03616a-kube-api-access-q9kvq\") pod \"must-gather-jhqwd\" (UID: \"d49c1137-0305-4431-9e94-37ef9c03616a\") " pod="openshift-must-gather-pplq5/must-gather-jhqwd" Nov 04 01:53:49.855553 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:49.855476 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pplq5/must-gather-jhqwd" Nov 04 01:53:50.135471 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:50.135384 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pplq5/must-gather-jhqwd"] Nov 04 01:53:50.139176 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:53:50.139144 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd49c1137_0305_4431_9e94_37ef9c03616a.slice/crio-8e638f2a3f8339bf6a68f958a69302b021adae3a2743870346dfc2beab8ddbb3 WatchSource:0}: Error finding container 8e638f2a3f8339bf6a68f958a69302b021adae3a2743870346dfc2beab8ddbb3: Status 404 returned error can't find the container with id 8e638f2a3f8339bf6a68f958a69302b021adae3a2743870346dfc2beab8ddbb3 Nov 04 01:53:50.142409 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:50.142391 5051 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 01:53:50.292625 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:50.292592 5051 scope.go:122] "RemoveContainer" containerID="9535e3f6520adc351dae74e5c5ccebe464720d836c88be0eeddfc8df0e0cdb72" Nov 04 01:53:50.373224 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:50.373189 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/must-gather-jhqwd" event={"ID":"d49c1137-0305-4431-9e94-37ef9c03616a","Type":"ContainerStarted","Data":"8e638f2a3f8339bf6a68f958a69302b021adae3a2743870346dfc2beab8ddbb3"} Nov 04 01:53:52.396251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:52.393436 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/must-gather-jhqwd" event={"ID":"d49c1137-0305-4431-9e94-37ef9c03616a","Type":"ContainerStarted","Data":"6122a00fb23ae391d25e3a00f590c098c74d1e9b98ad54f77cdb0c228878c0d9"} Nov 04 01:53:52.396251 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:52.393495 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/must-gather-jhqwd" event={"ID":"d49c1137-0305-4431-9e94-37ef9c03616a","Type":"ContainerStarted","Data":"ae139a03fc38e242841f64205f45d669daa143d97a696a1a8398ea8899149125"} Nov 04 01:53:52.411751 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:52.411703 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pplq5/must-gather-jhqwd" podStartSLOduration=2.157069318 podStartE2EDuration="3.411685342s" totalImagesPullingTime="1.254616024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 01:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:53:52.409701064 +0000 UTC m=+3371.381449936" watchObservedRunningTime="2026-11-04 01:53:52.411685342 +0000 UTC m=+3371.383434216" Nov 04 01:53:53.510050 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:53.509578 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8rwkl_04f82b22-3949-44d3-abe9-6473a2f68f76/global-pull-secret-syncer/0.log" Nov 04 01:53:53.550032 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:53.550005 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bw7wx_f4bef706-ca6f-4260-962f-ad0599ad093a/connection-checker/0.log" Nov 04 01:53:53.575856 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:53.575828 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hwnjk_ac81ee2f-5c6a-4ead-89c5-2a181b928494/connection-checker/0.log" Nov 04 01:53:53.602973 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:53.602948 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wmhkb_767984a4-597e-4498-a6bf-c988a7b228d1/connection-checker/0.log" Nov 04 01:53:53.799958 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:53.799878 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w6tt8_6ef08de9-97a2-4c5b-ae3a-66a9cd783a31/konnectivity-agent/0.log" Nov 04 01:53:53.863000 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:53:53.862967 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-48.us-east-2.compute.internal_60a4a4e0e49879528cdf55442dd62356/haproxy/0.log" Nov 04 01:54:01.502147 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:01.501660 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7457bc8fc9-m4sb2_d9622120-9f16-41e2-8a70-77f05616c703/authorino/0.log" Nov 04 01:54:01.581150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:01.580414 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tmmwg_c9f5857b-b55b-4002-b4a3-4f61950bd6ba/manager/1.log" Nov 04 01:54:01.581150 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:01.580572 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-tmmwg_c9f5857b-b55b-4002-b4a3-4f61950bd6ba/manager/0.log" Nov 04 01:54:01.750813 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:01.749960 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rztrb_f0dabfeb-1af7-4df9-9e2b-b96aa0b04d87/manager/0.log" Nov 04 01:54:03.277990 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.277733 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-kxzp6_d648b600-369c-47b0-ba02-6b3df1252bed/cluster-monitoring-operator/0.log" Nov 04 01:54:03.546836 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.546746 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p52db_c4850566-c180-40ed-b33a-45648462327d/node-exporter/0.log" Nov 04 01:54:03.588111 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.585399 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p52db_c4850566-c180-40ed-b33a-45648462327d/kube-rbac-proxy/0.log" Nov 04 01:54:03.610296 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.610269 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p52db_c4850566-c180-40ed-b33a-45648462327d/init-textfile/0.log" Nov 04 01:54:03.670635 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.670598 5051 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq"] Nov 04 01:54:03.678669 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.678640 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.692330 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.692302 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq"] Nov 04 01:54:03.774685 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.774649 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-proc\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.775006 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.774983 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-sys\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.775315 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.775287 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-podres\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.775400 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.775345 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-lib-modules\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.775462 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.775402 5051 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bnhv\" (UniqueName: \"kubernetes.io/projected/b30467c1-a6cb-4c1f-af1f-774cab106c7e-kube-api-access-8bnhv\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876280 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-proc\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876362 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876323 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-sys\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876401 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-podres\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876421 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-proc\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876436 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-lib-modules\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876473 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-sys\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876496 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-podres\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876514 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b30467c1-a6cb-4c1f-af1f-774cab106c7e-lib-modules\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.876616 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.876581 5051 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bnhv\" (UniqueName: \"kubernetes.io/projected/b30467c1-a6cb-4c1f-af1f-774cab106c7e-kube-api-access-8bnhv\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:03.885633 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:03.885600 5051 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bnhv\" (UniqueName: \"kubernetes.io/projected/b30467c1-a6cb-4c1f-af1f-774cab106c7e-kube-api-access-8bnhv\") pod \"perf-node-gather-daemonset-rwqtq\" (UID: \"b30467c1-a6cb-4c1f-af1f-774cab106c7e\") " pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:04.003933 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.003897 5051 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:04.057382 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.056568 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-q5894_526a942d-b2a6-42f6-a48f-761bdb2f73fd/prometheus-operator/0.log" Nov 04 01:54:04.096512 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.096465 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-q5894_526a942d-b2a6-42f6-a48f-761bdb2f73fd/kube-rbac-proxy/0.log" Nov 04 01:54:04.125263 ip-10-0-137-48 kubenswrapper[5051]: E1104 01:54:04.125225 5051 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wtpbl" containerName="extractor" Nov 04 01:54:04.130671 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.130585 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-cpht7_2da3265c-ae95-4220-a9d0-11ab44432db6/prometheus-operator-admission-webhook/0.log" Nov 04 01:54:04.294713 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.294679 5051 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq"] Nov 04 01:54:04.300566 ip-10-0-137-48 kubenswrapper[5051]: W1104 01:54:04.300538 5051 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb30467c1_a6cb_4c1f_af1f_774cab106c7e.slice/crio-3e1a5ba0f4bb67f7ee9705c3df73d6261b5bfa9a9e26ac218d5674861736e01c WatchSource:0}: Error finding container 3e1a5ba0f4bb67f7ee9705c3df73d6261b5bfa9a9e26ac218d5674861736e01c: Status 404 returned error can't find the container with id 3e1a5ba0f4bb67f7ee9705c3df73d6261b5bfa9a9e26ac218d5674861736e01c Nov 04 01:54:04.574236 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.574189 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" event={"ID":"b30467c1-a6cb-4c1f-af1f-774cab106c7e","Type":"ContainerStarted","Data":"d997af96e2768c32f68688202e523b11f8b892bb22d71ca0ad213fbeb431ad2d"} Nov 04 01:54:04.574445 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.574260 5051 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" event={"ID":"b30467c1-a6cb-4c1f-af1f-774cab106c7e","Type":"ContainerStarted","Data":"3e1a5ba0f4bb67f7ee9705c3df73d6261b5bfa9a9e26ac218d5674861736e01c"} Nov 04 01:54:04.575232 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.575210 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:04.596847 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:04.596784 5051 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" podStartSLOduration=1.596764632 podStartE2EDuration="1.596764632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 01:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 01:54:04.595312507 +0000 UTC m=+3383.567061379" watchObservedRunningTime="2026-11-04 01:54:04.596764632 +0000 UTC m=+3383.568513504" Nov 04 01:54:05.609945 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:05.609908 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-mh5d6_261f5fa6-5956-4a5d-a104-982dc13aed39/networking-console-plugin/0.log" Nov 04 01:54:06.218424 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:06.218337 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/1.log" Nov 04 01:54:06.220842 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:06.220706 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-7vtl7_9747f7d7-39f9-452d-9e9a-dbc9de48c89e/console-operator/2.log" Nov 04 01:54:06.778524 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:06.778486 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-7wfdb_112a671a-1e5f-4fe3-bc77-deb979fc4566/download-server/0.log" Nov 04 01:54:07.436576 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:07.436538 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-pv57c_e3ae027a-ff71-49e3-9a23-4e5ea2b6a6cd/volume-data-source-validator/0.log" Nov 04 01:54:08.627852 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:08.627824 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvtzk_b0807c2d-b756-42e8-86b3-97c30f23df72/dns/0.log" Nov 04 01:54:08.646918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:08.646889 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvtzk_b0807c2d-b756-42e8-86b3-97c30f23df72/kube-rbac-proxy/0.log" Nov 04 01:54:08.777162 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:08.777129 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-shjsf_51ff9909-e56c-4d27-be31-574bf33a8f75/dns-node-resolver/0.log" Nov 04 01:54:09.392808 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:09.392776 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b666b4847-lb44f_4145ecab-462f-4574-a346-61d7da286302/registry/0.log" Nov 04 01:54:09.420447 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:09.420419 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dcksb_5b7cc6df-efb4-4ed5-a299-99d3eafd8e56/node-ca/0.log" Nov 04 01:54:10.516746 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:10.516709 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6_6d63414f-bab7-4ddc-b3fe-aa0da66e7881/istio-proxy/0.log" Nov 04 01:54:11.605510 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:11.605464 5051 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pplq5/perf-node-gather-daemonset-rwqtq" Nov 04 01:54:11.696695 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:11.696661 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2mvtq_5f694985-9c0e-45ab-93b4-fa5ebe379290/istio-proxy/0.log" Nov 04 01:54:11.795067 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:11.795029 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bk4g2_8d8804da-7930-4bee-9769-6910cf9db442/istio-proxy/0.log" Nov 04 01:54:11.967446 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:11.967340 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/0.log" Nov 04 01:54:11.969521 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:11.969493 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/1.log" Nov 04 01:54:12.001456 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:12.001415 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xkmtt_ed9a741a-6c41-4981-a205-f01665ad111e/istio-proxy/0.log" Nov 04 01:54:12.130064 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:12.130029 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-669749fcf7-vqftg_7d024224-166d-4cb4-976f-75e0b47addae/router/0.log" Nov 04 01:54:14.420610 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:14.420563 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vl6w6_8dbd7a56-ab21-413c-abac-36b1f79f452e/serve-healthcheck-canary/0.log" Nov 04 01:54:14.824918 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:14.824881 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6_6d63414f-bab7-4ddc-b3fe-aa0da66e7881/istio-proxy/0.log" Nov 04 01:54:15.069822 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.069788 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-sgnvg_ede327f8-e1cf-4596-abe5-de379f528b15/insights-operator/1.log" Nov 04 01:54:15.072102 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.072074 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-sgnvg_ede327f8-e1cf-4596-abe5-de379f528b15/insights-operator/2.log" Nov 04 01:54:15.256157 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.256112 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtpbl_e7f2ccca-45df-41f8-bbf9-0b8aaa791160/kube-rbac-proxy/0.log" Nov 04 01:54:15.281538 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.281511 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtpbl_e7f2ccca-45df-41f8-bbf9-0b8aaa791160/exporter/0.log" Nov 04 01:54:15.311464 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.311436 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wtpbl_e7f2ccca-45df-41f8-bbf9-0b8aaa791160/extractor/0.log" Nov 04 01:54:15.356284 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.356249 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-cfwcb-f2q7x_608e1cfd-3e3f-4bad-b275-74805261379b/insights-gathering/0.log" Nov 04 01:54:15.397829 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.397799 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-czn8m-jxhgx_552c2135-598a-4a6a-a71c-0d3fac0ecae4/insights-gathering/0.log" Nov 04 01:54:15.549773 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.549704 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2mvtq_5f694985-9c0e-45ab-93b4-fa5ebe379290/istio-proxy/0.log" Nov 04 01:54:15.608749 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.608717 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bk4g2_8d8804da-7930-4bee-9769-6910cf9db442/istio-proxy/0.log" Nov 04 01:54:15.715019 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.714988 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/0.log" Nov 04 01:54:15.717371 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.717341 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/1.log" Nov 04 01:54:15.736287 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.736264 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xkmtt_ed9a741a-6c41-4981-a205-f01665ad111e/istio-proxy/0.log" Nov 04 01:54:15.810204 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:15.808620 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-669749fcf7-vqftg_7d024224-166d-4cb4-976f-75e0b47addae/router/0.log" Nov 04 01:54:17.950879 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:17.950841 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk_59ac90c2-5b91-45fb-a1b2-b443b86668d8/manager/0.log" Nov 04 01:54:18.006225 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:18.006192 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7d9c85cd8c-l2cbk_59ac90c2-5b91-45fb-a1b2-b443b86668d8/copy-manifests/0.log" Nov 04 01:54:18.518977 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:18.518947 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7csdwq6_6d63414f-bab7-4ddc-b3fe-aa0da66e7881/istio-proxy/0.log" Nov 04 01:54:19.229143 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:19.228203 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-2mvtq_5f694985-9c0e-45ab-93b4-fa5ebe379290/istio-proxy/0.log" Nov 04 01:54:19.295624 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:19.295588 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-bk4g2_8d8804da-7930-4bee-9769-6910cf9db442/istio-proxy/0.log" Nov 04 01:54:19.384444 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:19.384411 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-69d5854bb-87mwv_9d5e6ac7-97c0-407d-b050-bf4d8670b7e9/manager/0.log" Nov 04 01:54:19.388863 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:19.388836 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/0.log" Nov 04 01:54:19.390865 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:19.390839 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xfhf7_01677e74-8528-41df-aa1a-947aefe19645/istio-proxy/1.log" Nov 04 01:54:19.408706 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:19.408671 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-xkmtt_ed9a741a-6c41-4981-a205-f01665ad111e/istio-proxy/0.log" Nov 04 01:54:19.480670 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:19.480584 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-669749fcf7-vqftg_7d024224-166d-4cb4-976f-75e0b47addae/router/0.log" Nov 04 01:54:20.282165 ip-10-0-137-48 kubenswrapper[5051]: I1104 01:54:20.281926 5051 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ph6rn_11a8492c-3577-4e3c-b7ab-a8f84b5d8b03/manager/0.log"